Hamel and Robert Ta show that max reasoning on frontier AI models like Astra wastes tokens and money without better results. GPT-6.1 SOL hits 71.9% accuracy on DeepSuite at one-fifth Astra's cost. Medium reasoning often outperforms max, which can overthink problems and degrade performance.
But I've gotten really good luck by just going down to Astra Light reasoning.
Oh, really?
Wait, say more about that. And I get almost everything done, but with Astra Light reasoning.
Tell me more about the Astra Light reasoning and how you use it for AI builders so they could kind of understand.
Yeah, I mean, so like you have all the different models and codecs. I think Astra is the frontier model, like the most powerful intelligence one. And they have the reasoning levels like light, medium, high, max. Something else beyond that? I think like previously, if you're using these models, like using your coding harnesses, like a year ago, it was really common to just set everything the highest.
Yeah.
Let me get the best model with the highest reasoning level.
All the intelligence. Keep going. Think a lot.
Now with Astra, like it's very expensive, so you can run out of your quota really fast. And I started running outta my quota all of a sudden. I think a lot of people have that experience with Codex. You said you had that experience.
I had that experience. I just ran outta tokens today.
And so I was like, you know, I probably don't need all of this. Like I was just being lazy. So I was like, okay, like as a first step before going off of Astro, let me just set it to light reasoning and see what happens and
see how it does for
you. Most of the tasks that I needed to do, they were great. You can see even here on this chart, right?
Like here's Frontier Math
and you know, like it plateaued. Yeah. It seems like there's a plateau after a certain—
yeah, here's Terminal Bench Science. Yeah. Terminal Bench 4.0. Yeah. So we could see here on OpenAI's blog here introducing GPT-6 Astra for the AI builders out there that are looking to token optimize. Looking at Terminal Bench 4.0 as a benchmark in terms of, let's put it as output tokens. We could see like over here, GPT-6 Astra reasoning effort max, you get 56.7% accuracy and 72,000 output tokens.
See if they have a benchmark with GPT-6.1 SOL, which just came out today. I think there might be something like that. It would be
cool. I don't know. Here, let's Let's see if this comes up. Does it? Oh yeah, here we go. For all the AI builders out there, GPT-6.1 SOL just came out and you get near Astra intelligence, their most capable frontier model for a fifth of the price.
And so like, this is a good reason to just use 6.1 SOL and like at least try it on your workflows. 'Cause like, yeah, it's way cheaper than even Astra.
Totally. So Hamel, your point for AI builders out there that are trying to token optimize is really like, I mean, you even see it here, which is, hey, maybe don't like over-optimize your own individual harness or skills, but at the— as a first step to keep your life simple, because all these things are changing all the time. It's hard to stay on top of the meta and the latest on X or Twitter or GitHub. Yeah,
it's like basically you don't need Frontier— you might not need Frontier Intelligence for anything.
Yeah, maybe you don't.
Like, if you look at this, it might take a little bit of an ego. Yeah, like a hit to your ego.
Yeah,
but like maybe you don't.
If you look at it as well, GPT-6.1 SOL effort max, cost per task, $1.57, score 71.9% on DeepSuite, which I think is one of the most comprehensive, close to real-world.
Yeah. And you can see on here, like, so like this GPT-6.1 SOL is frontier intelligence even more. It's more powerful than Astra. I think it levels
out. Like if we look at the highest, it's near GPT-SOL. Cost per task, $1.57. On the max for Astra, it's about 5x that. So $7.50 for only 1.3 points more on DeepSuite.
Yeah. When I'm looking at this, like in this benchmark, like the second reasoning, highest reasoning level seems like a really sweet spot. What is that, medium?
Also, even looking, if you drop it from max and extra high, it scores higher.
So
on high, it
might be trying to overcook things.
From your experience as an AI builder, what do you mean by overcooking things? What do you mean by that?
Well, sometimes when we turn the reasoning level up too high, it just overthinks. It literally overthinks what it's doing.
Literally overthinking.
Because that's what you're telling it to do is just think more, think more. Things don't need that much thinking as you think it does.
Might be overcooking.
And sometimes it can hurt performance. And that's what it looks like happened here.
