Nick Perry explains ARC-AGI-3, a benchmark that tests whether AI can figure out unfamiliar tasks without instructions, not just recall facts. An AI gets scored on how many moves it takes compared to humans, who solve it in about 10. Perry argues this measures actual thinking, something most benchmarks skip.
Every AI benchmark you've seen tests what the model knows, not how it thinks. Because most tests hand the AI the problem and grade the answer. But ARC AGI-3 doesn't even give it the problem. Instead, it drops the AI into a video game with no instructions and no idea what winning looks like. So, it presses a button, watches what changes, builds a guess, and corrects itself until it figures out how to win. But the score is about speed. Now, a human beats the game in 10 moves, so if the AI takes 100, it scores 1%. Which is why 98.6% means the AI is nearly human in a world it's never seen. And that's the part nobody explains, because this test asks something no other benchmark does: whether the AI can figure out what the problem even is. Follow for more on what these numbers mean.
