Hamel Husain explains why RAG systems need separate debugging strategies. Evaluate retrieval like a search engine using precision, recall, and mean reciprocal rank. Evaluate generation with LLM-as-judge. Error analysis tells you which part is actually failing.
Your RAG app failed and you might not know where to begin. How do you debug this? So there's two parts that you need to be aware of. You need to understand what the R means, which is retrieval, and the G, which is generation. Retrieval problems should be evaluated and optimized like a search engine. You should look at the information retrieval regime and optimize your retrieval metrics, such as things like precision, recall, MRR, so on and so forth. Generation is where you have your more traditional eval approaches, such as LLM as a judge or codebase eval. Else, and those are two entirely separate things. Now, to figure out where your problem is, is it the R or the G, you should do error analysis. Error analysis is where you'll find out where exactly the problem is occurring so you can focus on either retrieval or generation. For more information on that, look in the comments.
