Hamel Husain says start AI evals by inspecting 10 to 20 model traces with a coding agent, not by building a formal evaluation system. Look at the actual data first. Glaring errors will tell you what to evaluate next, and external triggers like a new model release often force the work anyway.
We have a question from Lakveer, which is, software engineering is moving at breakneck speed. Agree. Lakveer feels like he's never had the time to do proper evals like he wants to. And he's wondering generally how you see this, like how much of not having time for this is organizational alignment towards having an eval policy first?
You know, I wouldn't get too academic about it or even try to get too dogmatic about it. Of like evals, you should do evals. So if you're not doing evals, you are in the wrong. You should feel guilty. No, like you should be doing evals if you need evals, if you want to do evals. Okay, let's forget evals for a second. Just start with looking at some data. That's all. Just look at some data. Even like 20 traces, even 10 traces. You can get your coding agent to help you look at traces really fast. They can have it render like a nice interface for you to look at those traces, help you sample those traces. And just look at 10 traces. Just start there. If you look at the data, it will motivate what you need to do next. You'll automatically figure out like, oh, there's some really nasty errors in my application that I didn't know about. How am I going to make sure that doesn't happen again? Or how do I like even solve this problem? I have no effing idea. how to solve this problem right off the bat. Okay, then maybe an eval would help you because you like work through that. But like, I want to make sure that people don't have this idea like evals is a separate thing, like this separate activity, like go do evals, I'm gonna go do evals. No, it's like part of your software development. It's part of— it's just part of what you're building. So like, at the point where you want to Start investigating, is your thing good? Like, if that's something that you care about, like, is it working? Then start— just start with looking at data and then the rest is motivated from there.
Totally agree. A forcing function is when a new model comes out, right? I've seen people forced to do this when new models come out and they're like, oh, I gotta switch to 5.6 or whatever, whatever, right? And then they're like, oh, how do I tell if it regresses on some stuff or works better? So even, you know, things like this can force people to do it as well, I've found.
Yeah. I mean, you don't necessarily need to go into like rigorous evals too early either. Like if you're just building some tool for yourself and you are the only user, then you are the eval for all intents and purposes. Like you don't need to do all this like stuff necessarily. You look at every example. And so there's certain situations where you don't necessarily need to, but okay, if you're building a product for other people, especially ours being used a lot, Or if it's being done in critical workflows or anything like that, then that's when you're going to want some idea of if it's working well.
