Robert Ta and Speaker B compare AI models at different reasoning levels, finding that maxing out reasoning can cost $7.50 for just a 1.3-point gain on DeepSuite and sometimes hurts performance. There's a sweet spot at medium or high reasoning before you start overcooking the model.
On the max for Astra, it's about 5x that. So $7.50 for only 1.3 points more on DeepSuite.
Yeah. When I'm looking at this, like in this benchmark, like the second reasoning, highest reasoning level seems like a really sweet spot. What is that, medium?
Also, even looking, if you drop it from max and extra high, it scores higher. Yeah.
Because it might be, it might be trying to overcook things.
From your experience as an AI builder, what do you mean by overcooking things? What do you mean by that?
Well, sometimes when we turn the reasoning level up too high, it just overthinks. It literally overthinks what it's doing, because that's what you're telling it to do is just think more, think more. Things don't need that much thinking, and sometimes it can hurt performance. And that's what looks like happened here.
