Claude 3.7 Sonnet Just Shocked Everyone! (Claude 3.7 Sonnet and Claude Code)

So today Anthropic finally released their new AI model, Claude 3.7 Sonnet, and there's actually a lot to digest because this one isn't your standard LLM, but rather a hybrid reasoning model that has a lot to offer in terms of the benchmarks broken. So if you actually take a look at what we see here, it says that Today we're announcing Claude 3.7 Sonnet, our most intelligent model to date and the first hybrid reasoning model on the market. It says right here that Claude 3.7 Sonic can produce near instant responses or extended step by step thinking that that is made visible to the user. API users also have fine grained control over how long the model can think for. And of course, this hybrid reasoning that they actually talk about is essentially where you have System one and System two thinking, which I'll explain a little bit in a second. And essentially this hybrid reasoning actually means that the model can offer responses that can be suited to both difficult queries and of course those quick queries where you need an instant response. So they talk about how they developed Claude 3.7 Sonnet with a different philosophy from other reasoning models on the market. So just as humans use a single brain for both quick responses and deep reflection, we believe reasoning should be an integrated capability of frontier models, rather than a separate model entirely. This unified approach also creates a more seamless experience for users. So they're essentially referring to this, where of course you have System one and System two. Of course with your System one, this is where you have your intuition and instinct. So Quite like humans, LMS can respond instantly. But of course with system 2 this is much slower logical, and this is of course where you think for your problems and come up with more complex solutions. And this is of course what they've embedded into the model. Now with Claude 3.7 Sonnet, you can actually control the budget for the thinking. So if you're a developer and you are thinking about the customizability for this model, you can actually tell CLAUDE to think for no more than a certain number of tokens. And it's up to you how long you want CLAUDE to think. I think this is something that is really valuable because of course it allows us to control how long CLAUDE thinks about its problems. Because one of the things I realized about the other thinking models is that sometimes the model will think about for our problem for maybe 10 seconds, when we wanted the model to think for maybe 100 seconds or even 200 seconds. So this is going to be something that is really useful. I will get to the very infamous benchmarks in a moment. But when we actually look at Claude 3.7 versus Claude 3.5 in the standard model Claude 3.7, this is just essentially a lot smarter than the previous Claude 3.6. And essentially this version of course you can just enable the extended thinking mode where of course it self reflects before answering, which of course improves its performance on math, physics instruction, following coding and many other tasks. And they generally find that prompting the model works similarly in both modes. So for those of you who do have prompts that used to work on Claude 3.6 sonnet, and it's quite likely that they will work on Claude 3.6 and will work the same way they have done on Claude 3.7, so there won't be any changes to prompting. Now, one thing that I found really interesting, and I am finally glad an AI company has done this, is now the fact that they are optimizing for real world focus. They state that in developing our reasoning models we've optimized somewhat less for math and computer science competition problems in and instead shifted focus towards real world tasks that better reflect how businesses actually use LLMs. The reason I think this is going to be game changing is because oftentimes we see companies focus and obsess over benchmarks that are in areas that aren't in everyday use. For example, when we are about to take a look at the benchmarks for Claude 3.7, you'll see that the benchmarks are still very impressive, but but a lot of those areas don't actually directly translate to world business use that the average everyday person is going to get value out of the model for. And I think this is why Claude 3.7 and Claude 3.6 have traditionally been better than ChatGPT and their rival counterparts because they train the models so that they're actually good for real world use and maybe not so much competition problems. So these are the benchmarks for Claude 3.7 sonnet and essentially right here we can actually see where Claude 3.7 sonnet exceeds and when one of the first things that we do notice is that Claude 3.7 Sonic isn't crushing these other companies in terms of the benchmarks. If you do remember, it was very recently that we did have the Grok 3 beta be released and of course we can already see that Claude 3.7 is on it. I honestly don't know how they did it, but for many and several benchmarks it does come out on top, for example in agentic coding and of course agentic tool use, which I'll dive into a little bit more. But these are the areas that are actually real world use cases. These, these two right here are really important. But in other areas it does seem like, you know, for example, visual reasoning and high school competition math. It does seem like a lot of these top models are kind of all converging towards the same area, around 86%. But of course, in the GPQA we can see that claw 3.7 sonnet manages to edge out over Grok 3 beta. Now I will say that this one is a little bit more interesting because of course I do think with Claude 3.7 Sonic it, it really is a model that you can't look at the benchmarks and judge it. It is a model that you truly have to use. And some of the tweets that I'm seeing in the AI community definitely show us that this is probably going to be the model that a lot of people immediately switch to. I wouldn't be surprised if Claude probably run out of inference for the model considering so many people already used the previous one. Now, like I said before, of course I don't want to focus too much on these benchmarks, but I'll show you the benchmarks that we do need to focus on. So this benchmark right here is of course agentic tool use. So this is the TAO benchmark. And this essentially is the framework that tests AI agents on real world tasks with users and tool interactions. So this is basically a benchmark that will actually have real world usage. And so this TAO benchmark here that we see is basically one that is really important because like I said before, this is something that is really, really needed for real world use case. So this, this benchmark basically just evaluates how consistently an AI agent can perform the same task consistently across multiple trials using a metric called Pass K, which basically looks at how reliably it does it over competed attempts. And I think this stuff is important because we need benchmarks that actually work for the real world use cases. Of course, competition math, the gpqa. This is of course great for assessing how smart an AI is, but you're going to need real world use cases if you want AI to actually be used in, of course, the real world. So by focusing on tool use, which is what this benchmark does, it basically, you know, looks for consistent behavior and it ensures that these AI agents are prepared for deployment in sensitive domains like customer service or healthcare. And of course, it wouldn't be benchmarks if we didn't get to the SWE Bench verified. So this benchmark is of course one that is for the software development niche and this actually achieves state of the art performance on the SWE benchmark. And this being state of the art really just goes to show how much better this model is over OpenAI's O3. We can literally see right here that whilst yes, there's a lot of deep seek hype, a lot of O3 hype too. Claude 3.7 actually manages to surpass those models by a pretty significant amount. And this is something that I don't just believe the benchmarks for this model, this is something that I've actually seen first case, when looking at people who currently code with claw 3.7 sonnet and all of the things that they're saying is basically that this model is pretty much outstanding. So we can see that OpenAI 03 is at 49%, OpenAI 01 is at 48%, Claw 3.5, Sonnet, all of them are basically around 49%. And then we get this massive jump to 62.3% and then all the way up to 70.3% with custom scaffolding. So that is a huge, huge jump. And then this one here I believe was the October 2024. So that is four months and we've seen nearly a 12% increase, which doesn't seem like a lot, but in terms of your actual day to day use, this is going to be a remarkably more helpful model for various different use cases when it comes to SWE related tasks. Now of course for those of you who may use Devon, you'll also see here that in the agentic coding evaluation, this model actually once again jumps up to 67%. So once again I think it's quite clear to see that the trend we're on we can see with GPT4, okay, starting at 49% and now with Claude Sonnet 3.7 already at 67%. Can you imagine where we're going to be in just a few years? I mean the future is truly, truly exciting. And if you're actually wondering about Claude coding, you actually may want to take a look at this. So this is basically a video where they introduce a new agentic coding tool that actually allows users to work with Claude directly in the terminal. Now this is something that is launched as a direct research preview to to enhance coding capabilities and features. Of course of this is basically where it can understand your code base, it can analyze a repository, it can provide insights into its structure, users can request changes, it can display the thought process it can generate and execute tests, resolving errors automatically. It can detect and fix build issues iteratively. It can push changes to GitHub with clear summaries. And this is a video that you're definitely going to want to watch. If you're someone that uses Claude for.

Coding, should we be doing Big smile or.

No, what you're doing, that's creepy.

That's sort of my.

I'm Boris. I'm an engineer.

I'm Kat. I'm a product manager.

We love seeing what people build with Claude, especially with coding, and we want to make Claude better at coding for everyone. We built some tools, one of which we're sharing. Today we're launching Claude Code as a research preview. Claude Code is an agentic coding tool that lets you work with Claude directly in your terminal. We're going to show you an example of it in action.

So we have a project here. It's a Next JS app. Let's open it up in an instance of Claude Code. Now that we've done this, Claude Code has access to all of the files in this repository. We don't know much about this code base. It looks like an app for chatting with a customer support agent. Let's get Claude to help explain this code base to us. Quad starts by reading the higher level files and then it dives in deeper. Now it's going through all the components in the project. Cool. Here's its final analysis.

So, say I was asked to replace this left sidebar with a chat history and I'm also going to add a new chat button. I'm going to ask Claude to help me out here. We haven't specified any files or paths, and Claude's already finding the right files to update by itself. Claude can also show its thinking and we can see how it's decided to tackle this problem. Claude's asking me if I want to accept these changes. I'll say yeah. Now Claude's updating the navbar, adding a button and icons as well. Next, it's updating the logic to ensure the saving state works correctly. After a bit, Claude completes the task. Here's a summary of what it's done. Let's take a look at the app. So we're seeing a new chat button and new chat history section on the left. Let's check if I can start a new chat while keeping the previous one saved. I'll try out the new chat button too. Great. It's all working.

Now let's ask Kuaud to add some tests to make sure that the Features we just added work. KUA is asking for permission to run commands. We'll say yes. COTT is making some changes to run these tests. After getting the results, it continues with its plan until all tests pass. After a few minutes it looks like we're good to go. Now I'm going to ask CLAUDE to compile the app and and see if we get any build errors. Let's see what it finds. COD identified the build errors and is now fixing them. Then it tries to build again. It will keep going until it works.

Now let's finish everything up by asking Quad to commit its changes and push them to GitHub. Quad creates a summary and a description of our changes. And it'll push the changes to GitHub. That's it. That's an example of what CLAUDE code can do. We can't wait for people to start building.

Now. There was also this benchmark that I forgot to include, but they've actually introduced a Claude benchmark for the model playing Pokemon. So it says here that Claude 3.7 sonnet demonstrates that it is the very best of all the Sonnet models so far. And playing Pokemon Red. Fortunately I don't actually play Pokemon, but they talk about that Pokemon is a fun way to appreciate Claw 3.7 Sonic's capabilities. But they expect these capabilities to have a real world impact beyond playing games. Because the model's ability to essentially maintain focus and accomplish open ended goals will help developers in a wide range of state of the art AI agents being developed. So that is why they did this. And I think that these kind of new benchmarks are going to be super entertaining and super interesting. So of course as well we do have this, which is of course something for the future. And this is basically where they said, and I quote, that Claude 3.7 Sonnet and Claude code mark an important step towards AI systems that can truly augment human capabilities. With their ability to reason deeply, work autonomously and collaborate effectively, they bring us closer to a future where AI enriches and expands what humans can achieve. And this is basically where they're stating that by 2027 Claude is going to be having pioneers. So first we having the assistants in 2024, then of course we have the collaborators in 2025. Claude does hours of independent work for you on par with experts, expanding what every person or team is capable of. Then of course in Claude 2027 this is where we have pioneers, where they're predicting that CLAUDE is going to be able to find breakthrough solutions to challenging problems that will have taken teams years to achieve, so the future for Claude is certainly bright. But let me know if you have used this already. Of course, if you go right here, you can see that Claude 3.7 sonnet is right there. And of course you can also see the thinking mode. You've got normal and then you've got extended. So it's completely up to you what you want to use. Hopefully you guys enjoyed the video and I'll see you in the next one.