Hamel Husain warns teams against swapping AI components like RAG databases or models without measuring impact first. Set up automated evals before making changes so you can actually see if they work, not just guess.
Your AI product isn't working. So what's a really common thing to do? Change your RAG database, change your model provider, change your agent orchestration, change your tools. That's just a prayer that is not going to go anywhere. What you need is evals. What are evals? Eval are a set of automated tests that you can run anytime you make a change. You shouldn't try new tools until you set up evals. Otherwise, you're just bike shedding.
