Matt Wolfe shows how AI companies game benchmark scores to look smarter than they are. Companies submit different models than their public releases, researchers delete hard questions, and one scientist admitted outright cheating. A recent article called a major leaderboard "a cancer on AI."
I've been spending time digging into how these AI benchmarks actually work, and I found some stuff that's quite shocking. We see all these benchmark scores and we think, oh, this AI must be better than that AI because number went up. But what if I told you those numbers are all kind of garbage? There's a major AI company that got caught submitting a completely different model to the leaderboard than what they actually released to the public. And then their former AI scientist publicly admitted, we cheated a little bit. I found some research showing that the scientists smartest AI models have learned to literally cheat on their tests. They actually delete test questions, rewrite what words mean, and hack the scoring system so they pass tests that are literally impossible to pass. And just this month, an AI company published an article calling the most popular AI leaderboard, and I'm quoting here, a cancer on AI. So today, I'm going to show you exactly how this works, who's doing it, and why. You should probably stop trusting those shiny benchmark scores you see in every AI announcement.
AI News: Dots, GPT-6.1 Sol, Sonnet 5.5, Gemini 4, and everything you need to know
