Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain

These generic metrics are most of the time, not helpful at all. They're way too generic. They don't necessarily correlate with actual failures in your AI product and they don't actually mean anything. And it can be extremely destructive because, like, you know, people get seduced into this, like, okay, I could just plug in this dashboard to my system. You can get this dashboard of metrics and can tell me, like, how I'm doing. Then you kind of have this illusion of, oh, like, I checked the box, I'm doing evals and I am monitoring my system. In reality, like, you're not monitoring anything. Just wasted a whole bunch of time.

Welcome back to Chain of Thought, everyone. I am your host, Connor Bronston, and joining me is my co host, Aten Dry Osanyal, co founder and CTO of Galileo. Aten, as always, great to have you behind the mic with me.

Always great to be here.

Yeah, I'm excited for this conversation because it's going to be on a few topics you and I are particularly passionate about and which I hate to tell our audience what they've probably heard us opine about a few times because we have a special guest joining us who's been at the forefront of Applied AI, helping over 30 companies navigate the complexities of building and productionizing their products. Hamil Hussain is an independent AI consultant, a luminary in the eval space, and has worked with innovative companies such as Airbnb and GitHub, which included early LLM research used by OpenAI for code understanding. Hamil, great to see you. Welcome to the show.

Thank you for having me.

Yeah, it's absolutely a pleasure. Because we've chatted a couple times about product philosophy, how to approach AI products today, and you're obviously well known for your blogs, your courses, and a particular favorite of mine is your field guide to rapidly improving AI products. You get right to the core of what actually makes AI products successful in the real world, and that feels like the perfect place for us to start our conversation. You open that guide with this concept of a tools trap that many companies are falling into. Can you start by giving our audience a bit of an explanation of this idea and why so many smart AI teams are falling into this trap?

Yeah. So a lot of times when people think about evals or measuring, you know, the reliability or performance of their LLMs in terms of, is it doing the right thing for the user? The first thing that a lot of people's minds go to is like, okay, what tools can I use? Can I just like Abstract this entire thing away to some tools. Can I just buy tools? Like, can I make it? Not my problem? You know, is there some abstraction or something that I can use to, like, have it so that I don't have to worry about the accuracy of my AI product or the performance of it or if it's doing the right thing, like, maybe something can just figure it out for me. And, you know, until we get to like, AGI or something like that, I don't think that's possible. But, you know, this is where people's mind goes, goes towards. And so I think the number one question I get is, hey, what are the tools? And that's the wrong question. The question should be like, hey, what's the right process? Like, how do you, how do you evaluate AI? You know, getting into tools later? But what is the right process to go through? Because no matter what tool you use, you have to go through a certain process to evaluate AI correctly. Yeah.

And Optin, I know you have a ton of thoughts about that process. And both of you, I've seen discuss this idea of generic metrics not being enough for many companies. You know, fancy dashboards being a panacea and, you know, a band aid solution, not something that actually solves everyone's problems. And this idea in your guide, as you put it, Hamill, of creating a false sense of measurement and progress, or as you've described, that Aten, you know, AI has a measurement problem. Hamill, could you give us an example of how you think vanity metrics have led teams astray?

Yeah, so it's really tempting if you're building AI tools. And Aten probably can provide more color around this. But I've seen this with other vendors. Not trying to pick on Galileo or really anyone, to be honest, is when you go into a pitch meeting around, hey, we can help you with your evals. You know, people want to see a solution and the. They want. You know, it's easy to kind of present a dashboard. It's convincing to some extent, if you don't know any better, to present a dashboard with all a bunch of generic metrics. Hallucination score, toxicity score, conciseness score, you name. Just so happens that these generic metrics are most of the time not helpful at all. You know, they're. They're way too generic. They don't necessarily correlate with actual failures in your AI product, and they don't actually mean anything. And it can be extremely destructive because, like, you know, people get kind of seduced into this. Like, okay, I could Just again, going back to the tools discussion, I can plug in this dashboard to my system. I can get this dashboard of metrics and can tell me like how I'm doing. And then you kind of have this illusion of, oh, like I checked the box, I'm doing evals and I am monitoring my system. But in reality, like you're not monitoring anything. You just wasted a whole bunch of time. You don't really know what your failures are and like what the most important things you should be focusing on. And so it just creates a lot of churn. And you know, I think people are getting a lot better with recognizing that now. But I think, you know, especially six months ago, it was the cause of almost all of my consulting business with people that were confused and hit kind of a roadblock or a wall in terms of, okay, we plugged in this tool, we got this generic dashboard thing and we don't really know what to do now.

No, I absolutely second what Hamel is saying. In fact, I would go to the extent of saying that this generic metrics problem has existed in machine learning even before Genai, even erstwhile ML workflows, you usually have a held out test set, measure F1 scores and just say the model is good or bad based on that score. Those approaches as well are akin to generic approaches where they treat all kinds of errors the same way. In some situations they're necessary, but they're in no way sufficient. And these were also some of the realizations that I personally had as well before even starting the company. When we built Michelangelo at Uber, there was no one stop metric that would be the panacea for your problems. And the same patterns are emerging again. And I'm curious to ask you, Himel, what kind of patterns you're seeing. But basically, just to take an example, with agents, customized architectures are kind of the way to go. You can build agentic architectures in a million different ways. And customized architectures need customized personalized evals which also need to evolve as your application grows and evolves and meets the new kinds of data. So one good question to ask, I think for a practitioner, for a developer, is rather than, oh, what metrics do I need from a buffet of metrics, rather, what are the pains or potential risks in the workflows of my app? Let's list them down and then author evals which are customized to those pains and then constantly monitor those pains because those pains will also evolve. Pains as in potential risks and pitfalls in your application and then accordingly update your the set of evals that you're using based on those. Those evolving pitfalls. But I'm curious to know if you've seen similar patterns. Mel.

Yeah. So one of the things that's really important to do with evals is to ground it in your failures. So how do you know what your failures are? And like, the thing that we harp on a lot of and what we teach and evals and what I write in my blogs constantly is look at your data. But what does look at your data mean? Look at your data. So what's behind look at your data is this process called error analysis. And error analysis is. Has been around for a really long time, even before machine learning. So it's like been around in social sciences. I recently learned that. I thought, you know, the first time I was exposed to it was machine learning, of course. But it is a kind of this process where you go through and you look at data and you take notes about what is going wrong, and you then use those notes and you kind of categorize them. You say, okay, like, what kinds of errors am I seeing? And you can do. It starts very simple, like, counting those categories and seeing, like, okay, what types of errors are happening the most. And then you make a decision, like, what to prioritize from there. And it's a very powerful technique that most people don't do because no one has taught them to do it, I think. And it's very simple. It's like the most simple kind of thing we like, you know, we're talking about like, opening a trace viewer and then like writing notes and going through a bunch of traces. And like, you know, there's some, like, okay, the same questions always come up, like, how many traces should I look at? So on and so forth. And there's some useful heuristics. There's this concept from social sciences called theoretical saturation, which is means like, hey, keep looking at traces until you're not learning anything new. So what we teach is like, try to look at at least 100 traces just as a heuristic to get people started because they have a lot of anxiety if you just say theoretical saturation. They get. They don't even begin. They just get scared of the whole process. But a hundred is like concrete number of people can know, like, have a goal. And then like, you know, after you begin, you don't really care about the hundred. You're like, oh, I'm learning so much. I think that's the counterintuitive part of, like, going through individual data points and reading what is happening like in a focused session provides immense value and people don't know that until they do it. And they're very surprised that about amount, you know, at the amount of value that it provides. And so that that can inform all of your evals activity. You know, it'll like, it'll motivate everything, like what you should focus on, what you should write an eval for, et cetera. And it's not really like this error analysis kind of like kind of bucket it in this activity of evals, but it's not even evals, it's like just development. So I'll just stop there.

That's super fascinating. I actually kind of, as you were talking, I'm drawing parallels to certain sort of opinions that we make on the Galileo platform itself, because we are an evals and observability platform is this new notion of quantitative insights or metrics and qualitative. And the qualitative bit to me sounded very similar to the theoretical saturation workflow that you're describing, which is the error analysis process where it's less about numbers between 0 and 1, measuring low and high, and it's more about more abstract. It's at a much more abstract level. Where are you achieving what you were set out to do? And along the way, what pitfalls or errors are you seeing? Something we do in Galileo is kind of drive the developer or the user to using what we call log stream insights. And log stream insights are more qualitative insights on hordes of your data, like segments of your long running sessions. Whether it's like a chatbot session or any kind of long running agent, we would analyze data in bulk and give you qualitative insights and then try to correlate them to potentially having you build some quantitative measures based on those qualitative insights. And hopefully the more qualitative insights you find you reach that theoretical saturation that you're talking about. So I can draw a lot of parallels and it's very fascinating to hear kind of the theoretical sort of side of error analysis and the practice of it being much beyond AI and machine learning.

I'm curious if the two of you think that part of the reason this approach to error analysis hasn't really truly been popularized in current AI development circles is because we've seen this change in Persona where most of the people who were doing machine learning work like yes, there were engineers involved, but it's a lot of data scientists who have kind of more classically been trained on some of these error analysis techniques. Whereas Software debugging is a different approach often. And we're now seeing kind of the marrying of these two approaches with engineers who are now becoming AI engineers and working very differently and having to transform both how they think about the software they create from deterministic to non deterministic, and also having to think about their approaches in different ways. Is that what's driving this kind of gap, you think, or is it something different?

I think so, yes. I mean, I would say the first epoch or phase of AI engineering was very much focused on, okay, like we need to build stuff, we need to go to 0 to 1 really fast and let's see what's possible in a rough sense. And you know, now that you know. And so it was very much the narrative and you know, also the truth. Like, you know, one of the most important skill to get started was software engineering, you know, in that, like, you need to, you know, glue together a lot of things, use APIs, you know, kind of full stack engineering, really important. And when it comes to, okay, like, how do you know that this stochastic system is reliable? That's a whole different skill set that takes time to learn. And, you know, there's a very large intersection between machine learning, data science, and the skills you need to do evals often. And the reason, you know, I tried to actually see how, how much I could get get away with in terms of like teaching engineers evals without data science background, or, you know, the requisite, let's say background. And you do hit a limitation really fast. And like, for example, you know, Shreya and I are teaching a class on evals. We've taught over 700 students so far of all different kinds of backgrounds. And you know, like, for example, when we get into building LLM as a judge, what we teach people is like, okay, one of the things that's important with LLM as a judge is that you can trust the LLM as a judge. And to trust LLM as a judge, you have to compare it to some human labels. And there's things like the questions always come up such as, hey, like, why is it okay to sample data? How can you know? And like we, we show people, like, okay, if you want to know how much noise there is in your judge, you can do stuff like bootstrap, bootstrap sampling. People don't understand that. They're like, why is it okay to like, just continue, like sample a whole bunch of times from a data set to get the distribution. And so we found that we almost have to go back to classic statistics and teach people that which is not super tractable, to be honest. Not in the format of okay, let me teach you evals real quick. You can teach fundamentals and you don't necessarily need all that stuff to get started, but you need a fair amount of data literacy. That's one side of the equation is like statistics, but also it's all the analytical tools, right? So like how do you, how do you like dig into data? You know, like let's say we're talking about traces earlier and like clustering those traces or navigating them or analyzing them. Like you want to be able to like really pick at data really fast and just do open ended exploratory analysis on it. And a lot of those data skills come into play again when it comes into like digging into a problem. And so like you very quickly arrive at the very similar skill set of a machine learning engineer or a data scientist. You don't necessarily, you don't need to be training models, but I would argue that you shouldn't be sending most of your time training models anyways. Like, you know, you were looking at, you were doing a lot of error analysis and debugging and whatnot. So that's my spicy hot take, perhaps in this podcast.

I don't think it's a hot take at all. I think it's a very, very legit take on just the distinction between software engineers and data scientists and answering that key question in the new world of sort of meshed roles and the AI engineer and what is mostly like technical people are kind of undergoing this minor identity crisis. And the answer kind of lies in what you said, which is if you were to cherry pick one skill that's needed for the software engineer to become the AI engineer or to be efficient in the modern era is really just the skill of understanding data and knowing the difference between good and bad data, or how to take bad data and step by step move it to good data and just data literacy is how you put it. I think that is the main skill because there's the other skills which are knowing the semantics of a decision tree which is totally commoditized and you don't need to know, you don't even need to know how to train models or fine tune them, but to be able to understand this basic process of comparing an output with a pre generated ground truth which, which is either human labeled or synthetic, but just knowing the goods and bads of the practices, that is what data literacy is. And if this skill is adopted by a tier A software engineer. I think they've set themselves up for the future, definitely.

And there's a lot of like, related skills as well, like designing metrics and the list goes on. Like, you know, how to, you know, tell stories with data, how to have a sense of, like, when your metrics are leading you astray, all the way down to, like, having good product sense and having that be aligned with, with metrics. You know, potentially doing AB tests. The whole suite of things is, is important. My friends and I joke that we might have a new job title coming called AI Scientist, but I try not to be the one who is coining you.

Wait a second, we're talking about AI, pms, AI engineers, AI Scientists now. Oh, man.

You know, there's always, every time there's a technological shift of some kind, there is kind of this sort of gravitation towards the idea of a unicorn. So we saw it actually like many times. The most recent time we've seen it is actually in data science itself, where initially at the outset of the data scientist, we had the person that did everything, software engineering, statistics, DevOps, so on and so forth. I think people realize there's a little bit too much surface area, honestly, and then kind of split it up into different kind of sub disciplines. I think we may be seeing that with AI engineer if I were to predict.

Speaking of AI engineers, I know one of the recommendations that you've made, Hamel, has been that when teams are making AI investments, particularly when AI engineers are helping make their decisions here, it's really important just to have a customized way of viewing their data, not necessarily a complex dashboard, so that they can approach this debugging as error analysis in the right way, so they can make decisions in the right way. Because as I think, as Aten and I have certainly experienced working with folks, it's very easy to overwhelm teams with too much data instead of enabling focus. Why do you think giving everyone an easy way to see what their AI system is doing is more impactful than some of the sophisticated analytics that I think often we're trying to reach for?

Yeah. So the guidance there is like, okay, there's a lot of tools out there that provide a good way to get started, like Galileo, like, you know, you have a way that you can like plug in your AI application and see your traces in a stream and kind of go through that. A lot of times in, you know, in your applications, there are a lot of domain specific things going on. Like you might have widgets that you're rendering your application Might be writing emails. You might have external data sources that you need to reference to evaluate a particular trace you might want to have. You might want to view the trace in the exact way the user is seeing it. For example, you might have things in your trace that, by default, are usually not helpful, but that take up a lot of space in terms of tokens, all kinds of, like, little nuances. So what you want to do is really dial in the data viewing experience so that you can do this error analysis and, like, reviewed lots of data really fast in a way that is very customized to you, that is very contextualized to how you want to see data. All the data you need to see in one place rendered in exactly the right way. And so the reason that's my. That's our advice is just because of AI, because, like, AI, you can vibe code. So, you know, AI is really good at producing simple applications that can render data and like, you know, have simple. Yeah, simple web applications like render data where you have, like, input fields and stuff like that. That's something that is probably, you know, below the bar where AI can clear. Clear those tasks very well. And so because of that reality, we recommend that people, in a lot of cases, like, create their own data annotation apps, because there's just way too much value to be had relative to the cost of doing so. It's not the case like, 100% of the time, but it's the case like, a lot of times.

Atit, I know a big part of our recent product philosophy at Galileo has been to give people more simplified views, whether it's the graph view or timeline view, which we've kind of designed with the idea of, like, okay, let's give them other options to debug agents. In particular, as we look at these kind of more complex systems, as well as other reviews that may or may not be live by the time that this podcast launches. And I know this is something you're thinking about a lot. This is something you're thinking a lot about, too, because as I alluded to, we've kind of had conversations together with AI engineers, I think, just like Hamill has, who are going, hey, I need help focusing here. I'm not really sure what to look at necessarily. I'm not sure where to spend my time in that error analysis. What's your philosophy on how to approach this? I guess observability and focus layer that Hamill is talking about.

Yeah, I think beyond the graph views, which is a feature that we offer features like graph views, kind of tend to point to the broader philosophy of giving the right abstractions to the user to be able to kind of do the segmented root causing of these ever growing sophistication in systems which have evolved from simple rag to agentic rag to multi agent. You want to give the users the right abstractions so that they can shine the torch in the right areas. And that's where views like the graph view, session views, interaction views, these come in to be able to give the tools to the user, to just be able to root cause effectively. And what that means, what that entails is you run your application end to end and each request may sort of touch certain parts of your application and light up the nodes there. And each request will run through a different sort of path in your application, which you can visualize as a DAG or a workflow. The first step is to be able to spot the anomaly where kind of the ground level customization on the metrics as well as the qualitative insights come in. But then these right abstractions and the right views to be able to make sense of what's going on. And then there's the data that's associated with, because all this is really is just data flowing through a bunch of nodes and edges. So once you spot the anomaly, you want to look at the data and what went wrong with that. So simplifying the views around the data is kind of the next step from there.

And just to be clear, like what I described does not at odds with these things in tools. They're just like supplementary like. I also always want a trace viewer like the ones in Galileo, because this can be like a lot faster to search through that and just look at that without. You know, sometimes I'm looking, sometimes I'm looking for something that maybe by accident wasn't in the annotation tool or something else. So it is really useful. And also like a lot of these platforms like Galileo have APIs where you can connect your annotation tool to and you know, write data back and forth to it. So, you know, that's just something to think about.

Yeah, and I think we all agree that's a great best practice is to leverage the APIs of whatever evaluation tool you're leveraging. Obviously we, we hope that scale, but whichever eval tool you're using, like using that API to bring that data into other places where you can look at it in different ways and kind of consume that information and highlight it to business users, I think is a fantastic thing to do. And Hamill, I know, you've talked about this idea of empowering domain experts who may not be in an eval product every day to add their insights and help improve these non deterministic systems. How do you think about writing and iterating on prompts with domain experts versus with engineers?

Yeah, so one of the biggest failure modes I see and is also one of the biggest drivers of my consulting business is people outsourcing evaluations to developers. Which is fine if you're building a developer tool where the developer is a domain expert, but usually they're not. And the, the kind of the symptom there are the root cause of people outsourcing evals to developers because they're thinking of AI, like software engineering. They're like, oh, it's a, AI development is a software engineering task. The moment you say anything about AI development process, they're like, this is outsourced to developers. That turns out that always goes really badly because yeah, like you're only guessing, you know, and the developers don't have enough context. So you want to involve the domain expert. So like, you know, if you're working building something for lawyers, you want to involve the lawyer, you want to involve a legal expert at some point. And so you know, when it comes time to doing things like iterating on prompts, you shouldn't have the prompt so removed from the domain expert. The whole point of LLMs is like humans can talk to computers. And so if you obfuscate everything so much that the domain expert can't talk to the computer, then you're kind of burning the whole value proposition of AI to begin with because you want a direct line of communication between your domain expert and your in like what's going into the AI in terms of prompting. And so what I described in that blog post is a lot of like a good pattern that I've seen work really well is if you have a user facing application, you know, have like an admin view where you expose the prompt and allow the person to change the prompt. Even if you don't want the user to change the prompt. You have the like for your internal purposes. You have an admin view that allows the domain expert to change the prompt and fiddle with it. It gives them like a more direct connection to what exactly is happening rather than like having conversation, abstract conversations about AI. And it should do this and it should do that. It's really important that they get in there and they are like experimenting.

Yeah. And I think it very much aligns to what Galileo has done with our continuous Learning through human feedback feature because we feel the same way. You need to leverage this domain expert feedback. You can't simply have. It'd just be the engineers who may be depending on your business, divorced from the bare metal of what the product's doing. Hopefully they are very aligned to that. But sometimes they have business users who are translating key pieces of that for them, or domain experts who bring a lot of context. And I know it's part of why, especially when we're looking at custom metrics. But all of our metrics we leverage feedback from SMEs, whatever type they may be can go in and say, okay, let me get feedback on these 10 traces and say, hey, this metric feels a little off. Actually this is pretty accurate. Or here's a little contextual feedback and then use a judge to translate that and apply it and retune the metrics. Something we're finding a lot of success with. But I think there's a lot more opportunity to go deeper here. To your point, it feels like too often, even in highly customized evaluation systems for enterprises, we are just scratching the surface of the human context that we can bring in. I mean, it's a very common problem for many organizations that there is too much tribal knowledge that's not living in documentations that's not necessarily making its way into systems. And to your point, it's so necessary that we bring that human knowledge into our AI systems because they perform best when they have the data they need. And it can be as simple as friction between technical teams and understanding that domain experts have of some of the jargon of your AI systems? Like you gave this great example in one of your pieces about translating reg to just making sure the model has the right context and really saying, hey, let's just put this in a term that anyone can understand, even if they're not deep in AI. What's your advice for AI teams who are looking to bridge that gap and really bring their domain experts into the fold so that they can be part of improving their AI systems and their AI data?

Yeah, let me clarify last point with some like concrete failure modes, like to look out for like one is, okay, there's a this, there's an aspect of like a prompt store or like a centralized place that you can put prompts, which is fine, but a lot of times what happens is folks don't build like properly enough around, around that they don't build like an experimentation environment. And so like you have to change the prompt there and then like commit it and then wait and Then like go somewhere else and like try something and that's like way too much friction. So that is kind of, you know, that prevents the domain expert from experimenting. A lot of tools have prompt playgrounds which are great. It's a good place to get started. However, most prompt playgrounds, they don't have access to your tools and your infrastructure and your application code. So they can't call, they can't perform rag and they can't call tools and they can't do all the things that your application is doing. And so, you know, you can't necessarily rely on that either. That's why you need this like integrated. I forgot what I called it in the, in the blog post thing called like integrated integrated prompting environment or something. Try to make up a name for it. Basically it's can you need to be able to play with the prompt in your user facing application directly? Because that's the only pattern at least that I've seen that's worked reliably in terms of bringing the domain experts in.

Yeah, I'll just add a couple of points here. First, of course the need for easy to sort of easy to use human feedback is critical. And some of our, yeah, like to your point Connor, some of our human feedback features which go much beyond, you know, just offering like binary signals, thumbs up, thumbs down, and the ability to create your own sort of feedback, kinds of feedback becomes important. But to Hamil's other point about just managing the prompts and offering the subject matter experts the ability to tweak the prompts to interact with the app, I think engineering wise the matter gets a little bit tricky. Especially for more sophisticated applications like multi agents where things are not necessarily driven by one prompt, might have a series of prompts which are triggered one after the other. You don't have control over many of them, but more often than not it's, it is driven by a kind of a seed query which is kind of the natural language interface to any gen AI app. So the engineering challenge kind of becomes how do you abstract the entire application and make it available in front of the user through a natural language interface, the user being the subject matter expert, not the developer, but being able to actually run the developer's app seamlessly so that to the SME it's all about here's my input. I have pure knowledge about my input and the expected output, but all the machinery in the middle you should be able to abstract out for me. So the trickiness kind of comes in the fact that I guess the challenges around how do you use our APIs and the SDKs and of course all the containerization technology to be able to kind of simulate a version of the app which may be a distributed app. It might be running on two different availability zones for that matter. It is just software. So I think that's where the challenge comes. And we are kind of at a point where it's doable to simulate sort of singular monolithic applications and make this workflow available to the SME. But it gets challenging when the app itself becomes distributed. And that's where kind of the, a lot of engineering innovation is going.

Yeah, it's really non trivial like you have to be, you have to think like you often can't like expose everything to the SME. You have to say is there a high value thing I can expose? And you know, it just, if anything else, it just helps give them intuition so they don't think rag is a very abstract concept or prompt is even an abstract concept. You know, you'll be surprised like how many people think prompt is an abstract concept because they say something in a meeting and the expectation is a developer is going to write the prompt. That's the worst thing that can possibly happen. So whatever way possible you needed to get away from that.

And so what I'd love to close the conversation with. And Hamill, thank you again so much for joining us. It's been a distinct pleasure having you. Is just some advice. Like what, what would be your summation, your advice to a team that is looking to build their eval system, that is looking to improve their AI product? What would you tell them?

Yeah, so the biggest two kind of things that I can think of is like one error analysis, also known as look at your data. It just, yeah, it solves so many problems. I was like maybe 90% of the whole evals process is, is like looking at your data. Like you find so much even before writing evals. Like you just find, you'll just find so many bugs, so many things, opportunities for improvement, so on and so forth. And then the next thing that I can think of that makes a huge difference is having an experimentation mindset. And this one you have to cultivate a little bit. There's some talks that I can point you to about how you might reframe your thinking. I mean this is something that's innate to machine learning folks and data science folks is like you don't have this waterfall chart of how to build a machine learning system. You have an idea of different experiments you want to try. You don't even know if it's going to work, but what you do have is a hypotheses of like, hey, this might work, this might not work, let's try this, let's look at this afterwards. And so you have to reorient a lot of things in order to do that. You have to kind of, you know, have a different language that you talk about within your teams and sort of make sure that you're not don't have those rigid approaches when it comes to this. Yeah, that's probably another podcast, but those are my thoughts.

I mean we can definitely have you back for another conversation because I think there is so much more we can go into here. Aten, how about you? Any closing thoughts from your side of the house?

Yeah, I would say that, you know, erstwhile before LLMs, AI was considered garbage in, garbage out. And now with LLMs, AI has become software. So software 3.0 is AI and now software is garbage in, garbage out. So to Hamil's point, do look at your data because of garbage in, garbage out. And secondly, I would say that there's three specific things that I've learned as kind of the layers of AI reliability. The bottommost layer is the kind of the brass tacks set up basic monitoring, traceability. That's just stuff that we've solved, you know, before AI happened. Traditional observability is a partially solved problem and there are certain things that are done well there. Adopt those practices. The second layer of the three layers is set up your prompts and your metrics and consider them as your evaluation assets. They're your first class citizens. They will evolve over time, have disciplined versioning lineage around them, set up a good system. There's and the third is the insights layer which is the whole qualitative insights turned into customized quantitative insights. So if you practice these three things and kind of consider them the three pillars of your AI reliability, you've you've built a good 360 evaluations and observability layer in your software.

And I'll add one more thing is take, take my course. So it's shameless Plug take the evals course. It'd be a good way to learn about how to get set up with evals.

And I'll second that and say also check out Hamill on x and on LinkedIn where he shares a lot of fantastic content. We will certainly link both those in the show notes. And yeah, Hamill's blog as well is a great place to go learn. Hamill, thank you so much for joining us on the show. It's been a pleasure.

Yeah. Thank you.

And to our listeners, if you want more fantastic content from Hamill and many other thought leaders, make sure you subscribe to the podcast because we share information from industry experts, perspectives from AI luminaries and hot takes, plus much more both in the podcasting app of your Choice and on YouTube. So whether you want to watch the conversation, listen in, or check out any of our other content here from Galileo, you can find us all over the Internet. We appreciate your support. And Hamilton, thank you again for joining me today.

Thank you. Thank you.