Hamel Husain covers three ways to deploy evals: CI/CD integration to catch regressions, online monitoring on live production data, and guardrails to block bad outputs before users see them. The right choice determines whether evals protect your product or slow it down.
In this 12-part series of evals, we've talked a lot about how to do evals. But once you already have those evals, what do you do with them? There's 3 different ways that you can deploy evals. The first one is CI/CD. So CI/CD is there to help you catch regressions. What you want to do is curate a set of test cases, have those test cases run in the same way that you run CI/CD normally. The next way to deploy evals is online monitoring. And online monitoring is there to help you catch real-time errors. So you can run your evals against your production traces, but also do other things like A/B testing and monitoring product metrics and user behaviors. The last way to deploy evals is guardrails. And guardrails are there to prevent bad outputs that would be catastrophic if your users were to see them. And you deploy all of these differently. This cheat sheet gives you a breakdown of all of those. This is the last part of our 12-part series on evals. I'm really curious what you would like to see next or what other kinds of series like this that we should work on. If you're building an AI product, think about the last time you changed your prompt. Did you have a good way of measuring if what you changed improved things? If not, evals are the highest ROI skill you can learn. Below, I'll share a link to our upcoming cohort where we will teach you how to implement eval workflows used by the top AI teams, as well as give you lifetime access to office hours so you can get unstuck.
