Meet Claude 3.5 Sonnet: First Impression of a model Superior to GPT-4o

We finally have a GPT4 killer, at least on benchmarks, I'm talking about the newly released Claude 3.5 Sonnet, which is the best state of the art model available today from Anthropic. Now, at least on Vision benchmarks, it's the state of the art when it comes to image understanding. And they also released something called Artifacts, which is a new way to use and interact with Claude, which makes it very useful. I'll show you a demo later in the video, but this is going to really change the way we interact with LLMs. Now, unlike GPT4O and Gemini from Google, Anthropic was able to release Cloudsonnet today with all the features and you can start using it right away. Now, when it comes to performance on benchmarks, it's probably the first model that is able to beat GPT4O as well as the original GPT4 model on almost all the benchmarks. This is supposed to be the second best model in the Claude family. We are not talking about Claude Opus 3.5, that's going to come later in the year. But if you compare the performance of Claude 3 Opus to Claude 3.5 Sonnet, the new model is going to be able to give you twice the performance, at least according to their own internal testing, with twice the inference speed and much lower costs. And the model is also available through their API, so you can start using it right away. Okay, so let's first look at the blog post in which the model was announced and then later on I'll show you how to interact with the model and how to use this new Artifacts feature, which is pretty amazing. So Claude 3.5 Sonnet is going to be available for free on Claude AI. They also have an iOS app and if you sign up for the Pro plan you will be able to get better usage. Now this model will cost $3 per million input tokens and $15 per million output tokens. In terms of cost, it's substantially lower than the Clothes Opus, but seems to have much better performance compared to the previous best Anthropic model, and this is evident from their internal testing. So on internal agentic coding evaluation, the Claude 3.5 sonnet solved 64% of the problems outperforming Claude 3 opus, which solved 38% of the problem. So if you look at it, it seems to have almost twice the performance of Claude 3 opus, at least on agentic coding tasks, which is pretty impressive. It seems to have much better image understanding. This is also clear from the benchmarks that they have Shared again on most of the Vision benchmarks. It's able to outperform the existing best models including GPT4.O and Claude 3 Opus, which is pretty impressive. Let's talk about Artifacts, which is a new way to use claude and I'll show you a demo as well. It's a new feature that expands on how user interact with claude. When a user asks Cloud to generate contents like code snippets, text documentation or web designs, these artifacts appears in a dedicated window alongside their conversation. So if we look at an example here, basically when you start the conversation you will actually see another window that appears where you can see the code running, very similar to code interpreter, but you can interact with the outputs that the model is generating. And we will generate a game based on this. It's pretty impressive to work with. Now keep in mind the Claude 3.5 sonnet is just one of the models that are going to be released. So later in the year they're going to be releasing Claude 3.5 haiku as well as Claude 3.5 opus. So I would suspect that the Opus is going to be much better compared to what we are getting today. And usually these 0.5 family just kind of gives you a hint of what the next generation of models are going to look like. So probably they're going to be releasing the 3.5 family and soon after they will probably release the Cloud 4 version. So just to show you how much progress we have made over the last year, here's a very good representation of AI model releases and their capabilities in terms of a timeline. GPT4 was released in early 2023 and it was the best model out there for think whole 2023. Claude 2 was released somewhere in July 2023, but it was a much less capable model compared to what GPT4 was able to do then. The Claude 3 opus, which was the first model that was able to surpass GPT4 Turbo, at least on benchmarks, was released probably two or three months ago. And now we have the cloth 3.5 sonnet, which is the current state of the art model, at least on benchmarks. And you can see that in the last few months there has been a lot of activities. So there are new Gemini models as well as GPT4O was released along with the Gemini Pro 1.5. So a lot of a lot is happening and I would suspect that OpenAI will probably release something very soon. Okay, so let's look at the artifacts from Claude 3.5 sonnet. Okay, so let's See how the artifacts works. And for this we're going to use the cloud AI. You can use this for free or you can sign up for the paid account which is $20 a month. Now here I'm going to select Claude 3.5 sonnet. Now in order to enable the artifacts feature, you need to go to your account and then click on Feature Preview here. Right now they have only one feature, which is the artifacts. Just make sure that it's toggled on to get started. I'm going to ask you to create a Flappy Bird game with scores, write HTML code and let's see what it comes up with. So the speed of generation is pretty fast, but you will see that it opens up a completely new window on the right hand side where it starts coding the game for you. So this is basically the artifact. You can interact with the artifact here. Sometimes you're going to see some issues, but then you can ask the clarifying things from the model. So you can see that it's not only generated the code, but it also generated a preview and you can interact with the game. I had to go back and forth with Claude for a little bit, but here is a working version and it just took us a couple of iterations. So I just had to ask it to adjust the effect of gravity as well as adjust the physics. And we have a working Flappy Bird game that Claude was able to generate for me without me writing a single line of code, which is pretty impressive. Now the good thing is that you can actually test the code that the Claude 3.5 sonnet writes right here using the artifacts. So it's a code interpreter but on steroids. I think it's a lot more useful because now you can interact with the code that the model is generating right here in the same window. Another feature that they are highlighting is its image understanding capabilities. So we're going to upload this image of Imad. And then I said who is in this image? Can you describe what you see? Now the response is I'm not able to identify specific individual in images, but I can describe what I see in this photo. So very standard behavior from Cloud Opus. So then it says the image shows a man sitting at what appears to be a podcasting or streaming setup. He is wearing a baseball cap with a logo on and glasses. And let's see if it able to identify the facial expression. So it says the man has somewhat surprised wide eyed expression with raised eyebrows. And now let me ask it whether it can identify what is on the cap. Okay. So let's see. It says the cap the man is wearing has a simple logo on it. From what I can see, it appears to be a stylized design that looks like two mountains or rectangular shapes forming an M, which I think is a pretty good interpretation. Then I said, what's on the cup next to him? There is a cup. I think it has the word impact on it, but let's see if it can actually make out something of it. So it says, looking at the cup on the table next to him, the man, I can see it. It's a black mug with some text or logo on it. Right. And the full text might be impact because it can only see the packed part and even from the image you can see that, okay, the M is visible. But yeah, this is pretty impressive because that is kind of occluded image now, so far, so good. I'm going to be creating a more detailed video on testing its different abilities, but at least we have a model that is capable enough that it's able to surpass GPT4, which has been a standard, or at least the GPT4 family of models have been a standard. So now we have a model which can beat that. And it's going to be available through the API as well. You can start using it today with all the features that are in there, unlike some of the other model providers. So really impressive stuff from Claude or the Anthropic team. Really looking forward to what they're going to be adding next because it seems like this feature preview part might have some more features in there. So it's not going to be only limited to the artifacts, but hopefully they're going to be adding a lot more stuff in there. So, really excited about the rest of the year, what we're going to have. I hope you found this video useful. Thanks for watching and as always, see you in the next one.