Lenny RachitskyHost of Lenny's Podcast | Author of Lenny's Newsletter

The biggest trap in evals

27 Sept 2025 · 0 min

Watch on YouTube

0:21

Lenny Rachitsky warns against automating error analysis with LLMs. They lack product context to catch subtle issues and bad product smells, often incorrectly saying traces look fine. Manual error analysis is essential.

Number one pitfall right here is people are like, let me automate this with an LLM. What we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be bad product smell or not. So I think in these cases, it's so important to make sure you are manually doing this yourself.