Jonathan and Robert Ta discuss why AI benchmarks matter less than testing models on your own code. OxAlpha scored 80% on DeepSwee, but Fable 5 reportedly beats it by 15%. They're building a custom benchmark for their codebase instead of relying on published scores.
OxAlpha getting an 80% reported DeepSwee score.
I'm gonna fire this up right after the pod. I see that score and I'm like, no way. I love Fable 5, my orchestrator. So for those that don't know, the DeepSwee score is supposed to be a benchmark that is more applicable and parallel to real-world development in production code. So when Fable 5 came out and me and Jonathan tried it, we immediately felt the difference of its orchestration capacity. And the DeepSwee score of Fable 5 has been just unmatched. And now this not only matched it, but went above and beyond by 15%, reportedly. I'm going to have to try that out. What do you think of that? 80% DeepSeek score? Like, isn't that wild?
Yeah, that's pretty legit. It definitely gets your attention. It means it's probably worth trying out. You know, I think if we really had our act together, we would have like a set of problems that we could test these models against. The new model comes out, we test it against those problems, and then it, it gives us like a grounded look at how well it performs at our work. That's where I want us to get to.
Let's add a ticket to our backlog. We're going to do this. We will come up with our own clarity benchmark. If you will.
Yeah. And how well it works for us. Yeah.
And how well it works for us in our production codebases and when we're doing customer work.
