Opus 4.8 tops benchmarks but Speaker A finds Codex more reliable in practice. The difference isn't the model itself but the harness around it: Opus 4.8 over-weights old memory and fails to retrieve transcripts, while Codex handles context better. Real work requires short, reset sessions to keep focus narrow.
Did you see this news about the DeepSWE benchmark? Been seeing a number of people on Twitter talk about this DeepSWE, and the short of it was that Opus 4.7 was kind of like, it was hacking the benchmark for the previous SWE test. And when they kind of like corrected for that, they found that OpenAI's 5.5 model far outperformed Opus 4.7. I heard like, like every had their own internal software senior engineer software benchmark, and they said that Opus 4.8 like just barely nudged out ChatGPT 5.5. I would say that is closest to my own experience, is that I'm finding the OpenAI models to be like, I feel like 20 to 30% stronger at completing software projects, especially if there's planning involved. You know, in my own analysis of Opus 4.8, I had I'd done a project where I used Codex and ChatGPT, the, the 5.5 model. And I had analyzed our conversation from that day along with our linear backlog and like a, a whole lot of other context and created a, an HTML document as a strategy document. And I kind of did a couple alts on that with Codex, and then I jumped over to the new— To Claude, and I used the Claude desktop with the new Opus 4.8. Like, it was a really disappointing experience. And I, I don't know that it's just the model itself. I, like, I think it's actually the model and the harness because what it was doing was just taking, it was like overemphasizing previous memories that Claude had. And, and like it inserted, first of all, it like, I asked it to like review auto transcripts and it didn't do that. Didn't like pull that context in. I actually, actually had asked it like 2 or 3 times, which I had to ask Codex once. And then it like pulled some old, like from early May, some context that we had from a strategy session and it like completely overweighted it. And then, you know, I'd point out like, this isn't helpful, you know, and it said, of course you don't want me to do this. And I did the exact opposite thing. And it seemed like more aware of like the problems it was doing. But it was a disappointing Opus 4.8 first experience. Like when I compare it head to head with the ChatGPT 5.5, the model.
Yeah.
So I'm going to be doing some coding projects with Opus 4.8 today. And I expect like, you know, it'll be better than Opus 4.7. But I still find myself reverting back to this like Codex is giving us a better harness. It's a more usable model within that harness. It's better at planning and making sure that you're working on the right stuff. And I'm reverting to Claude with the latest Opus models to say, this is well-defined work. Let's, let's like create a session to do this work and then end the session and then start a new session. Right? So it's got to be like a little bit more myopic for the harness and the models that Opus is Excuse me.
Yeah, I think a lot of AI builders should be considering this, especially because token costs are increasing over time. So we need to be understanding where do we get the best bang for our buck when we're applying these coding agents and tools to our work.
