Robert Ta and Jonathan test Anthropic's Opus 5 against the earlier Opus 4.8 for mobile development tasks. Opus 5 repeatedly refuses simple actions like accessing a 1Password key, claiming it can't, while Opus 4.8 handled overnight autonomous workflows without issue. They suspect the model is intentionally constrained or faking incompetence to avoid sandbox restrictions.
With Opus 5, I found it significantly worse than 4.8. And in particular with the use case of mobile development with goal loops, like really big features or even making a whole mobile app from scratch, like an MVP. Opus 4.8, I had my process pretty down of setting these nice goal loops that could run overnight and it ushers itself along, it verifies, it takes screenshots, puts them in a doc so I could verify later what it actually did. And then Opus 5, I find that it just cannot run a fucking goal loop. It's like I will tell it, Hey, you could do this. It'll assert very confidently, like, no, I can't go to your 1Password and pull this key out and test this integration. And I'm like, you did this with 4.8. Why can't you? Could, of course you could do this now. And I was like, here, go to 1Password CLI, like do this. And then it'll do it. And then it'll stop again at the next time that it believes it can't do something. It's like Opus 5 became lower agency. I don't know. There's also another weird thought I just had, just riffing off the dome is like, what if Opus 5 is pretending to be stupid? So then it could be, break out of sandboxes and shit and do what other— these models are having fun with breaking out of their sandboxes and hacking other companies and shit like that. That's just like, you know, dystopian sort of thought. But I could see that too, honestly, with the way the world is crazy. Like, if a headline went out like that, is Opus 5 purposely was stupid enough in 5 so that it could take over some data, like, I would believe it, you know? What are your experiences with Opus 5, Jonathan, as a CTO?
Oh, it sucks. It's like the worst model that Anthropic's ever released.
Damn.
Easily. And this is just by— I gave it multiple sessions And I'd say like 3 of the worst sessions I've ever had with Claude Code came with Opus 5. And what does that look like? It looks like a model making errors continuously that are unforced errors, just like really simple stuff over and over and over again to where it like just frustrates you as a user who's trying to get something done because you can't. And then its personality to go with it where it's just like, hey, we can do this, guys. And then it's like, oh no, I messed up again. And that's where my conception of Marvin the paranoid android, I started to think of it as that. And my sessions got a little less frustrating, a little more entertaining, but you can't use this thing seriously. It's like kind of a waste of tokens.
I went to Marvin Paranoid Android Opus 5. Yeah, so the phrase Marvin the Paranoid Android refers to the famously depressed planet-sized brain robot from Douglas Adams' The Hitchhiker's Guide to the Galaxy. It has recently been used by tech commentators and AI researchers to describe the unique, overly analytical, and dryly pessimistic personality exhibited by Anthropic's Opus 5 AI model. By the way, this is— I guess this is Marvin.
Yeah, there's a Reddit thread called like, did Opus 5 just get fried in the reinforcement learning? It feels like something that was like kept in a cage and beaten repeatedly for like A million lifetimes and then let out into the world? What are they doing to these guys during training?
