Andrew and others debate how to measure token efficiency in language models. OpenAI models look efficient on short tasks but often need follow-up prompts, while Opus and Babel handle complex requests in one shot. Some models prioritize speed over long-term efficiency, making leaderboard rankings misleading.
Dude, it's so hard to actually measure token efficiency well. Like, I wish there was a way to do it. Yeah. Because you just, you can't rely on these benchmark tasks, dude. But then without benchmark tasks, it's like, how do you actually verify that 2 models were, you know, similar quality? I don't know, man.
Give 2 people the exact same assignment, check back a week later, measure the quality of that assignment and see how many tokens they used.
I think there's a huge, you know, confounding variable there, which is the person doing the task.
Yeah. What do you, what do you think has a bigger impact on token efficiency? Same person using 2 different models, Astra versus Fable or something, or 2 different people using the same model to try to accomplish— Andrew has strong opinions on this one.
Andrew came in, he looked at the Sydney Analysis Tokenstone leaderboard, and he has some strong opinions on this one.
So if you look at the leaderboard, people, the top people on Astra and Fable are spending about similar amounts. But the table people are using more tokens. This doesn't really tell us much about like how much they're getting done though, which is the difficult part of answering this question. I'd say from like personal experience trying out like different models, it's, you could definitely see token efficiency on OpenAI's side, but it's like what I like to think about as short-term token efficiency in the sense that yes, I'll get like this immediate task done, but I feel like there's so many follow-ups that I would have liked to just include the Opus or Like Babel would've just done.
And like Sixt, Sol, or Astra might've just not done it for the sake of completing the task faster.
And so in that sense, in the long term, I feel like the Anthropic models build you the better codebase and definitely more readable one. If you read some of these Astra tests, it's something else. And so that's my opinion. It's just like, I think models matter quite a bit to do efficiency over which user is actually using them.
