The truth about Claude Code's evals

Unfortunately, X or Twitter is like a medium where you just get all these strong opinions of like, don't do evals. It's bad. We tried it, it doesn't work. We're Claude code and we don't do evals. And there's just so much nuance behind all of it because a lot of these applications are standing on the shoulders of evals. The fine tuned CLAUDE models have been evaluated on many coding benchmarks. They are actually probably very systematic about the error analysis to some extent. I bet you that they are monitoring who is using Claude, how many people are using Claude, how many chats are being created, how long these chats are. They're also probably monitoring in their internal team. They're dogfooding. Anytime something is off. They maybe have a queue or they send it to the person developing Claude code and this person is implicitly doing some form of error analysis. All of this is emails.