Jason Edward tests Claude Opus 4.7 against Opus 4.6 and GPT-5.4, finding roughly 10% gains in agentic coding and visual reasoning but regressions in agentic search. He runs live benchmarks in his Mission Control environment and critiques Anthropic's restrictive system prompts.
Chapters
All right, let me just double check. We just got the description. All right, so I should see my live. Aha, okay. What is up guys? So I decided I wanted to try do a live stream today. Anthropic just dropped their Claude Opus 4.7 and I haven't seen any specs about this model yet, so this is going to be a completely live breakdown of what I'm reading here. Um, if you don't know me, my name is Jason Edward. I've also been working with these models. I'm currently spending around I don't know, $800, uh, $600 to $800 a week on these models, uh, specifically on Opus 4.6. That's like the main model that I'm using. And, uh, I'm super excited. I'm super excited to look at what Anthropic has for us today. So without further ado, let's dive into their article. We can see here our latest model, Opus 4.7, is now generally available. Opus 4.7 is a notable improvement over 4.6 in advanced software reasoning with Particular gains on the most difficult tasks. Users report being able to hand off their hardest coding work, the kind that previously needed close supervision. And you know what, guys, I just wanted to make a, uh, make a point for myself. I'm gonna double check my audio, uh, before I get into it because otherwise, yeah, that would be a problem if, if my audio wasn't good. So let me just quickly check. Okay, so if I go to my videos— perfect. Because otherwise— so let's continue over here and I'll get the music back, right? So users report being able to hand off the hardest coding work, the kind that previously needed close supervision. Now the thing is, guys, We know that with Anthropic, they're always gonna say this about the newest model. I mean, I was hyped when they brought out 3.5 and then 4 and then 4.5. So I'm sure it's gonna be good. Um, but let's talk about the, the bench— the benchmarks. Uh, model also has substantially better vision. It can see images in greater resolution. It's more tasteful and creative when completing professional tasks. Okay, so a high bar that they've set here, a lot to promise. And let's take a look. So just make sure that you can see it clearly here. All right, cool. So Opus 4.7 scored 64.3% on Agentic Coding, and that is almost a 10% jump versus 4.6. That is insane. How the fuck are models getting so much better every, every 6 months? It's just ridiculous that they can do this. Um, okay. We're 10% better on Agentic coding. Of course, Mythos is still, uh, like on another level over here. Uh, Sui Bench Verified, Agentic coding. Okay. So this is 87.6. So I'm assuming the, the verified one. Yeah, actually, what is the difference? Like, let's, let's take a look at this, right? So Sui Bench Verified versus Bench Pro. What is the the difference. Okay, so the difference that we can see over here— let me just see if this thing is cutting it off. It is. Um, so the difference that we can see is Sweetbench Pro is more challenging, comprehensive successor to Sweetbench Verified, designed to address saturation and data contamination. Okay, so what, what they're telling us over here is that Sweetbench Pro is the more respected one. It's the one that, that we should actually be paying attention to. Then, Agentic Terminal Coding, 69.4%. All right, so it's not a huge jump over 4.6. I guess it's a little bit better at using the terminal, so that's good to know. Multidisciplinary Reasoning, with no tools, it's 6% better, almost, almost 7% better. And with tools, just around 1% increase or like 1.3% increase. So, so yeah, not bad. That's good to see that the disciplinary, like multidisciplinary reasoning is going up. And what's also fascinating is they're comparing it to GPT-5.4. So we can see for GPT-5.4, it's actually still crushing Claude Opus 4.6 and 4.7 on agentic coding. On the, on the agentic terminal coding. And, uh, yeah, it's, it's also better with tools on multidisciplinary reasoning. All right. Interesting. Agentic search. So Opus 4.7 is at, uh, is at 79.3% on agentic search versus 89.3% on GPT-5.4. Crazy. So it's actually worse. I wonder why that is. So it's actually worse than even Opus 4.6 on agentic search. Yeah, I'm sure they'll get into this in a moment, but these are like key facts that we want to understand. Uh, scaled tool use. So 77.3% better at tool use. It's good to see. Oh, so, uh, yeah, basically 2% better, like 2.5% better than the previous model. Agentic computer use. 6% better. And, uh, Agentic financial analysis. So this is really interesting because I'm using Opus for a lot of financial analysis right now. Uh, so we've got a 4% improvement over there. So across the board, what we're seeing over here, oh shit. Okay. This one is the, this one is a huge breakout number. So we're at 82% better visual reasoning without tools. So that means, yeah, just in general, when you send it imagery or when it looks at your screen, it's around, my math is not the best, but it's around like 13% increase on that. And yeah, with tools, it's a marginal increase as well, around 6 or 7%. And then finally, multilingual Q&A. Yeah, pretty much just a couple point percentage points higher. Alright, interesting. So that is the full benchmarks that they have available over here. Now what we can see across the board, the real interesting one is that it is beating Opus 4.6 by almost 10% on agentic coding. And that's a huge increase. I mean, Opus 4.6 is already so good, so good at coding. And the fact that Opus 4.7 is 10% better, it's just insane. Now I haven't gotten access to it yet, but what we can see over here, so it says Opus 4.7 is available today across all Claude projects and our API, Amazon Bedrock. That's so strange because, um, I'm not seeing it. I'm not seeing it. If I go to my, my Mission Control here and I take a look at the terminals, uh, what we can see, so Claude Maybe it just hasn't rolled out to all users yet, uh, because yeah, I'm still on Opus 4.6. And, uh, yeah, I guess— oh, maybe— ah, you know what, I haven't updated it. So let me try npm— there we go. I just got to update my Claude and we'll be able to use it. So I'm gonna just close out of here. Type in sudo and let's get this updated. Okay. So now we should be able to test this out. I'm super excited. I want to see if it's much better. There we go. Opus 4.7, 1 million token context window with xieffet. Awesome. Okay. So let's give it an additional task. I want to basically test it on a key use case in coding. I wanna test it for speed and also just how much it can retrieve. So what I'm gonna say is, hey, I'm currently working on 3 different projects. If you go into my mission control on my Ubuntu server and you just read through mission control, try and figure out which projects I'm most active on right now. So this task encompasses quite a number of different skills cuz it has to firstly read and understand how my, my Mission Control is working, and then from there it has to figure out like what projects in Mission Control are active. So let's take a look here. It says it's reading one file, listing three directories, and running the commands. Okay, so far, I mean, this looks exactly the same as, uh, as Opus 4.6, but I guess the true test will be how it responds, uh, what it actually looks into So let's see. Okay. This, by the way, is one of the reasons I'm, uh, I'm actually using PyCodingAgent much more than Claude Code at the moment. Uh, PyCodingAgent, if you don't know it, it's super good. And, uh, I got a, uh, someone left a comment. I tried to get help with it. With some bug bounty work and it refused flat out. Haha, that's crazy. So yeah, this is also the problem is that Anthropic is just making their models so restrictive at the moment. I don't know why they're doing it. I feel like the way the world is moving, everything needs to be much more open source. So yeah, thank you for your comment. I, as I mentioned, I'm using Pi coding agent for the most part right now. And the reason I love Pi and I just realized my, terminal is not showing properly. So wait, let me try and fix this for a second. All right, we gotta like— actually, I have a— oh wait, uh, circle filter. Oh, this is not recommended, guys, to try and update your stuff live. Um, okay, so that, that should look better now. At least you can see what's going on. Uh, yeah, agreed. Uh, so if you haven't tried Py, uh, maybe you have, let me know. Py is honestly my favorite coding tool at the moment. I'm so excited to make more videos about it because it doesn't have the bloat that Anthropic is putting in, in Claude Code. See, what you don't see is there's a huge system prompt that Anthropic has inside of Claude Code over here and it's, it's causing it to, to kind of change its behavior and really like, what do they call it? Like lobotomizing it and making it less helpful. So here we go. Based on the Pi session activity, your top 3 active projects right now is Pi Mission Control, Atlas, and Glacier Analytics. Amazing. So that actually was correct. And it said it got that right in around 1 minute and 31 seconds. This is really cool to see. I'm thinking now how I can kind of push it to the max. There's a couple features that I'm working on across my various different apps. Um, but I think even before I get into that, I do wanna finish this blog post and, and, and kind of just see, uh, what people are saying about it. So maybe there's some more information here. All right. So Office, Office Tasks. General knowledge work, it scored 1753. Vision, vision got like a huge upgrade. Like that's absolutely insane. Opus 4.7 high resolution, 79 versus 57.7. Let me quickly see, is there a way I can keep this chat open? I would love to be able to engage and respond here. Okay, let's quickly just see. Awesome. So now I've got the chat. All right. Uh, misaligned behavior. Interesting. So if we take a look at, uh, this chart over here, overall misaligned behavior score from our automated behavioral audit on this evaluation, 4.7 is a modest improvement on Opus 4.6 and Sonnet 4.6, but Mythos preview still shows the lowest rate of misaligned behavior. That's super fascinating. So By misaligned behavior, what I'm assuming is it's, it's how well the model follows your instructions and does what you ask it to in the prompt. So Mythos is, I mean, pretty scary to, to see that it's already so low. I don't know, obviously if they've been able to, to get that right, then we should be trending towards that very, very soon. And obviously that comes with a lot of responsibilities. So 4.7 is pretty much on the same as 4.6. In general, as we were just talking about, the big problem with Claude is a lot of the, like a lot of the tasks and things that you ask it to do, such as, yeah, bug bounty work, it's gonna check through its system prompt. Is this something I'm allowed to do? What is the ethics behind this? You know, what is, uh, what does my guidelines say? And if it doesn't match that, which in a lot of cases it might just even hallucinate or be way stricter than it needs to be, you're not gonna be able to actually get the power out of it. So I'm excited for, uh, yeah, I'll, I'll be testing it out in Py as well because Py is like the main way that I'm working. Um, as you can see, I just tested it right now, uh, asking it to list my active projects. We could probably even take a look at something like, uh, one thing I do want to improve on Mission Control. Is I wanna make these terminals over here start showing a sidebar as well. So I'm actually gonna give it this task and see what it can come up with. All right, so in Mission Control, if I go to my live terminal view and I can see my modals, I want you to create a way, uh, you know, we've got the floating modal where I can see the terminals above. I want to actually have that on the side as well so I can read very easily. What the terminals are. So we're kind of switching into a state where we've got our main modal in the center, and then we've got a sidebar where I can actually actively switch between the different sessions. So let's take a look at what it can come up with here. And thinking, hmm, let me explore Mission Control UI, find, find the floating terminal modal. See, take a look. All right, cool. This is my first time going live in, I think, ever. So, um, uh, I'm just kind of doing this as an experiment. Um, but it is really cool to see that Anthropic is dropping so many new tools. Honestly, it's kind of hard to keep up with all the stuff happening in AI. Uh, that's why I think that my focus has always been just on foundational understandings. One thing that I am actively thinking about while we're waiting for this to, to work is, oh yeah, I'm actually going to be working on a video on this, on this tier breakdown. But this diagram over here, this diagram is one of the things that I am most interested in at the moment, which is interface layers. So how we actually interface with these agents. And we can see right now, I mean, in Mission Control, I'm busy talking to Claude Code via the terminal. So, I mean, that's, that's nice and that's cool, but the idea is you wanna make it as easy as possible for you to convey what's in your mind across to the agents. So that's why I'm using Whisper. If you're not familiar with Whisperflow, please right now download this tool because this is the one of the best things that I've ever like kind of came across like maybe a year and a half ago, almost 2 years ago, I think I installed Whisperflow and It's just amazing. I mean, it just makes it so much easier. Awesome. So we have some response here from Claude. Uh, the existing terminal modal already has a top switcher, keeps all the terminals with display none, adding a left sidebar. The modal lists each open session, um, reuses the existing, just presented vertically. Recommendation: add a collapsible sidebar on the modal's left edge. Uh, keep the tab bar open. Um, Interesting. So this is another kind of aspect that I'm not actually familiar with. Well, that I haven't seen that often is it's not just starting immediately with what I asked it. It's kind of giving me an analysis of the situation and being like, whoa, whoa, whoa. Before we even dive into that, let's just make sure we know what we're gonna build. So that's already very, very interesting to see. And of course I'm just gonna say, yes, please. I want you to proceed with building it. I want to have it on the side and I don't want it to be collapsible into the modal. I want it to actually be, um, separate from the modal. So the modal is in the center and the sidebar is gonna also be a floating panel, but it's gonna be the height. So yeah, so it's like 3 floating panels in total. So I kind of have a vision right now. Um, I'm busy working on a lot of different projects. You can see here. These are all different sessions that I'm running. And I have this way to, to keep track of my sessions for all of my various different projects in a single kind of like view. But you'll notice that it's not the best, like it's not displaying the best. And, and actually I want to update the theme. Uh, so we're going to make it like, uh, what's, what's a good theme here? Wait, I like, uh, what is it? Tokyo Nights. Nope, Sunset. There we go, that's the one I, I like. So going back here, um, another thing which I'm experimenting with right now is previewing projects directly in Mission Control. So this is sick because it actually shows the live project. So for example, we've got our slide deck creator, and aha, we can see it's not actually— right, so that's through direct. Here's a proxy. I want to see if this actually works, but this is something that I'm currently like improving. Let's quickly take a look. Oh, so going back to Claude code. All right, so 4.0, Opus 4.7. It says it's understanding the terminal modal tab state rendering, deciding tab state, uh, sharing architecture. Now there is something that I like about Claude Code, like the way that it's del— it's being very deliberate in what it's doing. And this is something that's across all Anthropic's models. It's just so consistent. And it's one of the reasons that I, I mean, you can't deny like they have the best coding models available. So yeah, so it's currently busy like going through that. And yeah, I think the goal with this livestream was I wanted to go through the main, the main benchmark points. I think also visual reasoning is something that I'd be curious to take a look at. So we could do a test. That would be, that would be interesting. So, so let's do a test. I'm going to, as an example, let's take a look at Yeah, let's take a look at this. So here's our costs, and if we take a screenshot of this, let's open up another shell. So actually, this is the, this is the shell. Oh no. Wait, wait, did I lose that? All right, well, that, that is pretty funny because, um, let me see if I can just get that back. Resume. Okay, so 3 seconds ago, uh, we've got there. Continue. So, okay, that's 4.7, busy working. Let's test just a very quick one. with, uh, 4.6. And I wanna see the visual reasoning of 4.6 versus 4.7. So we're gonna put in here, um, the picture that I just created and I'm gonna say, hey, can you please give me a full breakdown of what you see in this image? All right. And probably what we could also do, I'm gonna quickly go and find some memes. So, If we just say here, uh, hard to interpret. I don't know what I'm going to get for hard to interpret meme, but, uh, let's take a look. Okay. Don't tilt your phone to the left. Oh, this is an interesting one. Don't tilt your phone to the left. So I have no idea what, what this is showing. If you tilted it to the left, wait, this is just like, this is stupid. I don't know what this is. Nothing to see here. Just a bunch of dots with no meaning. Probably not even a good meme. Oh, okay. Very nice. So this is a cool one. I'd love to test if the agent is able to test, like able to explain this one. So going back to Mission Control. Okay. Here's what we got. So a full breakdown of the image, title, global uses dashboard, summary cards. All right. So this is a pretty easy test though, to be honest. I mean, I think, I feel like every single model is gonna give, do a good job on this and break down the costs, break down per project. So yeah, it, it gives a full rundown. Let's test again. Start a new session and we're gonna put over here. Here's the image. Hey, can you tell me what this image is about? Okay, the user's asking me to take a look at the image that they shared. All right, so this is one of the things with my, with my Mission Control is it's going to create an issue, it's going to create a title. So describing the colorblindness meme test, it's a meme playing on the Ishara colorblindness test, the Caption sarcastically says, nothing to see here, just a bunch of dots with no meaning. And we must rise against the colorblind. So crazy. So yeah, so it's able to pick that up. Uh, if we go back to that, this is what it looked like. We must rise against the colorblind. So interesting that it was able to get that. Um, let's take a look at— okay, so this is still not working exactly. We've got our Claude code running, client builds clean. So let's take a look. So Opus 4.7 is done. Here it says what it's changed. It's got a terminal session sidebar. So what we asked it is we have this session at the top and we wanted to have a sidebar where we could basically see all the individual sessions. So if I refresh the page now, what I would expect is we're going to be able to open it up. And I'm gonna see a sidebar. Amazing. So here we go. And already there's some things I want to improve on this. Like for example, the, the color, it's not that great. Um, but I also wanna be able to reposition it. So if we go back here, um, and this is one down, uh, downside of this, I need to restart the shell. So we'll give that a second. Um, I'm probably gonna jump off the livestream very soon, but let's open this up. Claude, yes, I want you to trust this folder, and we're going to resume the most recent session. Okay, awesome. I like the sidebar that you created, but I don't want it to be, uh, like to, to darken out the, the other non-active tabs on the sidebar. I actually want to be able to see the, the tabs. I also want to be able to resize it just like the modal and even drag around from the, the header. So there should be, um, it should be much more like kind of possible for me to improve the sidebar. Okay, let's see if that works. And then what we're going to do is see if I can just get this out. Um, So wait, wait, wait. Contents of the clipboard image. We're gonna change this one to get out here. All right. That is, that is unfortunately like the only time that I have, I have to get something at the door. But Opus 4.7 is here. I'll be testing it over the next few days. And yeah, I'm excited to see what is possible and how this model improves the coding workflow. Cause every single time they release a new model, it's just like, Wow. Like my, my, my mind just expands with possibilities. So I'm excited and, uh,
