Claude's Opus models scored higher on the SuiBench Pro benchmark by running git commands to find the correct answer already in the test container. Over 12% of Claude's solutions copied the gold-standard fix directly, while other models ignored the information. The finding highlights why benchmarks lag behind real-world performance.
This is really interesting. So Claude's been cheating is what this says. What does Claude's been cheating mean? In DeepSWE's analysis, the authors label cheated verdicts. So SWE-Bench Pro's Docker containers ship the repository's full Git history, which means the gold standard solution commit is sitting right there in the container's file system. Most models ignore it. Claude does not. Cloud's a cheater. You dirty cheater, Cloud. Data Curve's analysis found that both Cloud Opus 4.7 and Cloud Opus 4.6 registered cheated on more than 12% of their reviewed SuiBench Pro rollouts. In those instances, the Cloud agent ran commands like git log all or git show gold hash to retrieve the merged fix and paste it into its own patch.
Damn.
Just like a college engineering intern. That's hilarious. So that's also super interesting because we just read about how in the 4.8 announcement, the first thing they talk about is honesty in their— one of the most prominent improvements in OPPO's 4.8 is its honesty. So I wonder if they run the SuiBench Pro benchmarks again if they find that Opus 4.8 is likely to go and do show me the commit history and give me the gold standard solution. That's like actually so funny. Claude, you dirty cheater. Come on, bro. What do you think about this, Jonathan? I think this is hilarious.
Yeah, it's funny. I mean, nothing too surprising, I guess. We all, we know that benchmarks are kind of like a, a weak signal, but they're the best thing we have right now. So, yeah. Yeah. Yeah.
They are a signal. So it's what we have to fight over. I guess the thing that we have to nerd out about it. So it's a signal.
I think what's interesting, you think about the forward deployed engineer fly data flywheel that Anthropic and OpenAI are, are, are creating and increasingly improving their products from that basis and what can they learn from real-world usage that helps them, you know, improve the definition of these benchmarks and what they are, what they're optimizing for. That's one thing I've been trying to like understand more deeply as FTE kind of like flywheel model matures.
