Hamel Husain explains how tools like Braintrust, Arize, and LangSmith let you version evaluations alongside your AI system as it changes. Early development is chaotic because both the system and its tests shift constantly, but in steady state, only the system changes while evaluations stay fixed, making comparisons meaningful.
The tools that we give you an overview in class, like Braintrust, Arise, LangSmith, they all have— there's more tools than that, but those are the ones we run into the most. They allow you to run evaluations, store the metrics, and version them. So you can version the evaluation. You might be changing your system, your prompts, your RAG pipeline, your tools, whatever. And then you might be changing your evaluations. Your evaluations will be updated over time. Your datasets will be updated over time. Anytime you change your datasets, You can version or use evaluations, either the LM judges, the datasets, or the evaluations themselves. You can version them because you can't necessarily compare evaluations. It can be a little chaotic at first if you're first building something. In the steady state, it's less chaotic when you're changing your system more than your evaluations. What happens is those tools allow you to see, so you can create, you have all your evals, whatever different judges, different codebase evals, and you'll have a suite of them. You can produce an aggregate score over them. A lot of them will be binary, pass or fail. You can have an aggregate score. You can rather Report out.
