Thew New "Claude 3.5 Sonnet" Actually SHOCKED The Industry! - Beats Gpt4o

So Anthropic just went ahead and actually shopped the entire AI industry because they went ahead and released a very fascinating model, Claude 3.5 Sonnet, which is now the current state of the art in terms of AI models, meaning that currently this is the best AI model that you can use and interact with on the planet. So without further ado, let's dive into what makes this so amazing. You can see on the screen that Claude 3.5 Sonnet performs better than any other model from any other company currently. And this is something that definitely caught us off guard considering the fact that GPT4O was released fairly recently. Llama 400 billion parameters is actually relatively really good. And of course, Claude 3 opus was the previous state of the art model competing with GPT4. Oh. So the reason that this is such a surprise and such a shock is the fact that this is not going to be anthropic's largest model. 3.5 Sonnet is actually the second model in the tier, which means in the future, when they release their updated model, it's going to be even crazier. So we can see here on the GPQA, which is graduate level reasoning, this actually takes a 6% jump over GPT4O on actually a 5.9% jump over GPT4O. And this is rather impressive. It also gets 88.7% on the MMLU. And you can see that on the coding, 92%. On the multilingual math, 91.6% and the reasoning over text, 87%. And the big bench hard, 93%. And of course on the math benchmark, so 71.1%. And the grade school math to GSM8K benchmark, 96.4%. Now, what's crazy about this is that I always like to look at the fine print of these benchmarks. And we can see here that for the majority of these, these are actually zero shot, which means that this is just one question followed by an answer. Now, in some cases where it does say zero shot chain of thought, that just means it's one question, one where they basically ask the AI system to explain its line of reasoning and then in doing so it gets a better answer. Whereas with these ones where you can see three shot, this is basically where they have three interactions with the model and then they ask it the final question and then that is how this reasoning is done. Of course, right here you can see that this is five shot, which is definitely very interesting because GPT four O's is zero shot. But I'm not trying to nitpick. This is definitely a very, very stark improvement considering that this model is blazingly quick. Now, the benchmarks aren't, of course, the things that we want to look at what actually got released with this, because they did show us a few interesting things that they're adding to Claude. So one of the things that they spoke about was the fact that this has strong reasoning. It sets new industry standards for the benchmarks. And we can see here in this demo exactly how Claude 3.5 Sonnet is able to help this user. Sam, you can see that they're helping them be able to craft this novel and of course, kick off the plot and give various pieces of information that are truly state of the art. Early tests from many users are stating that this is nothing like they've seen before. And you can see right here what's really, really intriguing is, is that Claude is outputting a huge amount of text in another box right here, which allows you to look at what the model is doing while it's actually generating the text. I think this is really, really useful because we can see exactly how the model is generating everything and it just gives it a cleaner user interface. Now, what's fascinating about this example here is that you can see that they actually ask the user to do quite a lot of things. And you can see here that, that it's making a detailed diagram of how relationships are and how they connect to the characters. That is something that I think is really, really cool. And it shows that Claude actually has some increased capabilities. One of the ones that they talk about, and something that I've heard as well, is that Claude has always so far been the very best at coding and interpreting what you want to do with your code. So for this, this now means that we do have a free coding model that is truly, truly advanced. And you can see here it's using its strong reasoning abilities in order to generate a response to the user's inquiry. Now, there was also something that they released that I found was really, really cool. Okay, and I'm going to talk about that in a moment. But they also speak about how CLAUDE now comes with stronger vision. This is a short demo on where they showcase exactly how CLAUDE is used to do many different things with its strong vision capabilities. You can see the user here actually asks Claude that I'm giving a lecture to my class on the human genome. He then inputs two images and says, this is a genome timeline of the sequencing. You can see the two images here. And he says, can you transcribe the data into JSON, then we can see that because Claude 3.5 is actually really, really quick, it's able to code this up very, very quickly and of course, efficiently without any mistakes. Now this is something that I love about Anthropic. They've released a product that is not only good, but it's actually really, really quick, considering the price to speed ratio is very, very effective. You could also see here that the user is able to add their color palette in. And then of course, once they've added their color palette in, it was able to make the adjustments and changes there. And then we were able to get this nice visualization where we can actually look at the data in a more efficient way. Then of course, what was really cool about this, and I think that this is going to be something that a lot of people do use, is that they create a presentation based on this topic very, very quickly. He says, use one of those charts you just used on the slide. And you can see here that within, I guess you could say a couple of seconds, we do get a quick presentation that we could use for pretty much anything. That is a workflow that we're seeing demonstrated by Anthropic that's rather effective on showing us how we can actually utilize Claude's new vision models. So by combining the different inputs, we can then ask Claude to make things for us, especially in code and in all these different formats, we're able to leverage these strong vision capabilities. And this is something that's really, really effective. And as the user was able to do this within only a few minutes now, there is also something called artifacts. So essentially artifacts are really, really interesting because they actually appear next to your chat and they allow you to see and iterate and build on your creations in real time. So the user starts by stating that, can you create an 8 bit style crab for me? And then you can see that it responds very quickly with the code on the right hand side. This is effective because now we can essentially do things that we want to. So for example, he says, can you make some seashells in the same style? And he says, absolutely, I'll create some seashells to complement our crab. Then of course you can see that those seashells are there. And then what's interesting is the next step. So what the user asks Claude to do is essentially build many different things within this artifact area. And you can see that as he's managing to build all of these things, we're then going to get to a situation where he combines all of these into a working game which looks very, very effective. So you can see right here, he says, let's make it playable, have the crab jump up over oncoming she shells. Maybe add some styling too, I think, to what I want to call this a crab Claude. And then we can see that this user is able to play this game after a few more lines of code from Claude. And and then of course what we're able to see is that this is something that the user can enjoy. So this is another demonstration of how the user interface in CLAUDE is changing. This is of course something you can try out right now, but I think it's rather effective that Claude is adding consistently new features, especially being a state of the art model. Now what really caught me off guard, and this is something that really caught everyone off guard with this entire model, is for the fact that of the price to intelligence ratio, what we can see here is the previous trajectory for the models for Anthropic in the cost per million tokens versus the intelligence. Previously we followed a traditional slope where as the cost was low, the intelligence was low and things sloped up very, very smoothly like this, as you would expect on any normal chart. But over time, interestingly enough, what we see right here in the middle here is this huge jump up in capabilities. For the same price, we can see that Claude 3 sonnet is actually the same price for a higher level of intelligence than Claude 3 opus. So this actually means across the board the cost of intelligence is going down. And this is a trend that many of you in the AI industry probably would have known as coming. But to see it so rapidly, considering the fact that Claude 3 Opus was held in such high regard to to be dwarfed by a much smaller model, I'm guessing, considering it's much faster and it also costs less, is something that I didn't think we would see just for a couple months. Now I'm really intrigued with as to how they've managed to achieve this in such a short space of time. But I do know that Anthropic are actually working around the clock to bring us new frontier models. Now, something that they also did actually speak about as well that I've mentioned before, was the agentic coding. This is that Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude Opus. And it says our evaluation tests a model's ability to understand an open source code base and implement a pull request, such as a bug fix or a new feature. Given a natural language description of the desired improvement. For each problem, the model is evaluated based on whether all the tests of the code base pass for the completed code submission. The tests are not visible to the model, which is important and includes tests of the bug fix or new feature. To ensure the evaluation mimics real world software engineering, we based the problems on real pull requests submitted to open source code bases and the changes involve searching, viewing and editing multiple files, typically three or four or as many as 20. The model is allowed to write and run code in an agentic loop and iteratively self correct it during evaluation and then we run these tests in a secure sandboxed environment without access to the Internet and the results are shown in table 3. So you can see here, overall Claude 3 sonnet is nearly as twice as good as Claude 3 opus on coding evaluations and you can see here at the other models how they face in comparison. So this is fairly fascinating because this sets a remarkable precedent for what is to come. Next we can see that Claude 3 haiku 17%, Claude sonnet 21, Claude 3 opus a stark jump to 38% and Claude 3.5 sonnet at a staggering 64%. That is pretty, pretty fascinating. I would have loved for Anthropic to show us some of these internal demos so we could see exactly how Claude is behaving during this agentic loop. Now, one of the main things that many people are wondering right now is that if Claude 3.5 Sonic is such a giant jump from the previous Claude 3 model, what is the jump for Claud 3 opus going to be like in the future? You can see here that they said our aim is to substantially improve the trade off curve between intelligence, speed and cost every few months to complete the Claude 3.5 model family. We will be releasing Claude 3.5 haiku and Claude 3.5 opus later this year, so it will be interesting to see how much Claude 3 Opus improves on its previous benchmarks. Remember, Claude 3 Opus was actually the state of the art model I until today. But now of course considering the fact that GPT4O was released which slowly dethroned Claude 3 Opus now that Claude 3.5 Sonnet is here, this is fairly surprising. If there was a state of the art model from Anthropic, I would have predicted it to be claud3opus since that was their largest model and their most capable model. Which means if Claud 3 opus or Claude 3.5 opus is coming in the future and it's going to be their most capable model, we can surely say that the benchmark improvements are going to be absolutely dramatic. And considering the fact that the other areas of competition and the other companies are also hot on the heels of Anthropic, it seems that this industry is getting no slowdowns anytime soon. Now, what they also spoke about in terms of the future models, they actually stated that they are working on new modalities and features to support more use cases for businesses and including integrations with enterprise applications. And one fascinating thing is that they said that they're also exploring features like memory, which will enable Claude to remember a user's preference and interactions that happened in the past, making their experience more even personalized and more efficient. So overall, what we have here is a fascinating showcase of exactly what Claude 3.5 sonnet is. This is a remarkable AR release, and Anthropic continue to surprise us with their ability to.