Carlo CocuzziAI Expert, Pod Cast Host, Real Estate Investor and Funding Specialist

Anthropic's Own Researcher Calls Out "Benchmaxing

3 Aug 2026 · 1 min

Watch on YouTube

1:24

In this brief YouTube excerpt, two speakers (A and B) discuss recent large‑model releases from Anthropic, focusing on Opus 5 and comparing it to the earlier Fable 5. Speaker B notes that Opus 5 shows strong benchmark performance, especially a dramatic jump on the ArcGAI‑3 visual‑code task (from 1.5% to 30.2%), and suggests the model may have been mildly optimized for front‑end development and vision‑plus‑code tasks. Despite these gains, B still prefers Fable 5, citing cost and personal experience. Speaker A echoes the same experience and points out a tension between subjective impressions and benchmark results, hinting at possible “benchmaxing” – the practice of tuning models specifically to excel on certain evaluations.

How strong is Opus 5 and how excited are you about it? I'm somewhat excited. I'm not over the moon.
I'm not as over the moon as I was about Fable 5. Opus 5, the benchmarks, the evals that demonstrate the strongest performance, and I don't think this is a coincidence, for example, ArcGAI-3, which is focused on the ability to solve interactive visual problems that humans find easy but AIs have historically found hard. It went from 1.5, at Opus 4.8 up to 30.2%, which is as of this moment last I was tracking, the highest official score from a baseline model. My overall whiff from using Opus 5 quite a bit is there was maybe mild optimization toward front-end development and anything that touches the nexus of vision and code. They're trying to lean in a bit to some of the gaps. Honestly, I still prefer Fable 5 even though it's more expensive.
So Alex, I have the exact same experience, 100%. But then I look at these benchmarks, and they seem to tell a different story.
How is that? I'm a little bit scared of that there may have been some mild benchmarking here. That's what I was politely gesturing at. The overall sort of distribution of tasks, definitely from interacting with it for a while, feels biased towards cogent and visual stuff and away from general capabilities outside that.