New Claude 3.5 Sonnet is Better Than GPT-4o

And it happened again. One of the big AI companies, in this case Anthropic, released a state of the art model in multiple departments. Claude 3.5 sonnets. And as opposed to some of the latest LLM announcements that were exactly that announcements, we didn't have anything usable. This one is a breath of fresh air. You can use it right away, even for free. There's limitations. We'll talk about this in a second. But not just that the benchmarks are better than even GPT4O, but supposedly the vision recognition is state of the art as of today. Meaning if you upload images, no other model is better at recognizing what you gave it. And this is my favorite. They made a massive step forward in terms of how the interface of an AI is presented to consumers. They introduced this experimental feature they call Artifact, where if the AI writes code for you, whether that's a website or a game, there is one half of the screen where you have the chat as you're already used to with apps like ChatGPT. But the second half is used as an interactive code code editor. And by default you don't even see the code, it just happens. It's interactive. So you talk to the chat on one side and you get the results on the other side. This is something we've seen with demos like Devin, but none of these were available to the public. They always just announce stuff and we never get our hands on it. GPT4.0, voice assistant, Sora, Devin and also many Google products are just a few examples. This is available today for free and especially this interface really is a glimpse into the future of these products. So let's have a closer look at everything that came out here with Claude 3.5 Sonnet and if you should be using this over something like ChatGPT with the GPT4O model. All right, first things first, the facts, what is happening here? Claude 3.5 Sonnet, a brand new model coming from Anthropic AI released today and it's their best model yet. And for all you true enthusiasts, this might be a bit surprising, right? Because up until now they had three models. Haiku, the smallest one, Sonnet's the medium sized one, and Opus, their big boy competitor to GPT4 and now 4.0 and there's no update to that large model. This is the medium sized model that got upgraded to the 3.5 model and it outperforms the their big model Opus. It has a whopping 200,000 tokens of context that you can use. And let's just briefly Talk about branchmarks. Yep, it crushes virtually all benchmarks against Opus and GPT4O. Nice. But as you know, I care more about the usability, the use cases for consumers rather than these benchmarks. But it's always good to see that they yet again raised the bar on what's possible. Another fact that this is fast. OpenAI kind of set the standard with GPT4O generating super quickly. Opus was slower than that. This is twice as fast as Opus and feels virtually identical to OpenAI's GPT4 model. If you had experience with that over the past few weeks, its knowledge cutoff is in April 2024, which is super recent. That's two months ago. And last but definitely not least, it's freely available to everybody, even in Europe. I'm sitting in Lisbon, Portugal and accessing this without a vpn. Now, do keep in mind that if you don't have a paid account, you're going to be limited to around 12 messages every few hours, which means you won't be really able to complete projects, but you can get a taste for it or get some quick results. For my testing and examples, I'm on the Pro plan here and now we can talk about the innovations here, because there's two big things I want to highlight. The fact that it's slightly better on some benchmarks at this point. As a consumer, I'm not sure if that really matters, as long as you're not talking about something like code generation or math problems. Because let's be real, if it's a 86 or 88 on MMLU, for most people watching this video, it's not going to make a difference. What is going to make a difference, though, is the vision capabilities, because the various jumps between the models recently have. Have been massive. Current state of the art GPT4O before that opus, but now Claude 3.5 sonnet smashes most of these benchmarks, especially when it comes to reading charts and documents, which is interesting because that's a very common use case, at least for me. I upload a lot of charts or infographics, and what this means is that even more complex infographics will be easily digestible. All right, but those are all numbers. Let's see practically how it performs. I went ahead and put this to the test right away. My first example is one that we've actually explored on the channel with previous versions of Vision, and it's this road sign that is quite complex. You need to look at all of this, you need to make sense of it, and then you can make recommendations. I basically uploaded it and asked can I park here on Tuesday at 6am? And it correctly answers that yes, you can park here on Tuesday on 6am and then there's the reasoning. Quick comparison. Inside of ChatGPT we get the same thing. Mind you, this already worked with ChatGPT Vision on release, so let's step it up a bit, shall we? I provided it with this image from a Where's Waldo book. You're probably familiar. They're very visually complex and you need to find this tiny Waldo character which has a red and white striped T shirt, pretty for humans, very hard, time consuming. That's why it's such a popular book, because it's a good challenge. I uploaded this image to both Claude and chatgpt and gave them the identical prompt. I told it where's Waldo? Question mark. Very innovative prompt engineering right here. I know. And I was surprised by the results. Both of them told me that, hey, to find Waldo, you can look for his characteristic outfit, but the scene is extremely busy and intricate and I can't actually locate him. Hmm, weird, right? Well, thanks for letting me know that this is a busy picture, but where is he? Where? Where? And Claude Sonnet continued to apologize to me that it can't actually find or point out where Waldo is. And let me tell you, no matter how you prompt it, it's just not going to tell you. And this is not because the vision is not capable, it's because Anthropic is notorious for its limitations. And what we ran into in this example is that they're refusing to identify various people in an image. In this case Waldo. It just won't do it because people could be abusing this for nefarious reasons. ChatGPT on other hand, when I told it, but where is he? Straight up told me Waldo is located near the bottom left corner of the image. I spent about three minutes looking for him, still couldn't find him. So I followed up with be more precise, I can't find him. And it gave me pretty exact coordinates and more descriptions. Where was he? There he is. And yeah, turns out there is Waldo. You can't even see the shirt, it's only his head, so it's more restricted. Just something to be aware of. Let's give it one more shot here. I uploaded the image of the first YouTube homepage that pulled up for me and then I gave it a simple prompt. List all the details of this image relevant to a YouTube content creator creator in the generative AI niche and the results are quite Interesting, because the category that Claude Sonnet 3.5 chose for me are very, very different from what ChatGPT decided to do with the same prompt and image subjectively, and this is just me and my personal taste, I would say that the Claude results were more relevant. Plus I can tell you something about the anthropic family of models. The writing style, generally speaking, and I think most people would agree, don't have this typical chatgpt style that is usually frowned upon these days, as in details that it was able to pick out from this. Both of them were excellent. I can't really make a valid judgment because both of them picked up on all the details that I would care about here. So look, at the end of the day, I think when it comes to vision, it really matters what you think and what your use cases look like. And one, you'll just have to try yourself and see how it works for you. And two, as time passes and more people test out their use cases, we'll learn more and I'll be sharing it here on the channel. But the vision looks excellent. And now let's talk about the second point, which is something that I absolutely love to see. It's this new experimental feature that you can enable by clicking on this little icon here at the bottom. And it's this Artifacts feature. And, and this is simply explained, the Claude models always excelled at coding. Now, according to the benchmarks, they're even better than they have been before. But what if you're not technical? What if generating code is no use to you? Well, a lot of people, including me, were trying to make these simplistic tutorials where, you know, generate the code anyways, don't read it, just copy paste it and bring it into another code editor that can display it. And you can still kind of work with it. But let's be real, for most people who don't know how to write code, this workflow doesn't work. And even for people who write code, it is cumbersome to always take it, bring it over to your ide, run the code, and then having to copy paste the feedback back into something like claude. That's why certain code editors like Cursor, where all of this happens in one interface, have become so popular. But now they're bringing all of these capabilities to the consumer. You don't have to know how to code in order to do things with code, because with this artifacts feature, you can do something like this. Generate a portfolio website for a designer. That's the simplest prompt you could come up With Right. Very intuitive. And it's going to start writing the code and you're going to be like, okay, great. Now I have a window with a bunch of code. I still don't know what it means, but. But wait a second. As soon as it's done, it switches to this preview mode and you have the viewer right inside of this interface. This thing is interactive. You can press buttons, you can move around and you can have multiple projects open in one. And if you want the code, you can always switch on over here and copy it or download it for yourself. But if you don't want to mess with that, you don't have to because the chat is still here on the left side. And as I mentioned earlier, this has been a thing with applications like Devin that have been announced previously but but never released to the public. If you're not familiar with a similar interface, chat on one side, code editor and preview on the other. Here they combine them. So now in practice I might generate a website like this. But hey, this should be a portfolio website for a designer. This looks like something that a 10 year old that is learning to code would create. Do 10 year olds learn how to code? I guess with apps like this they will be Anyway, besides the point, if I don't like this, I can follow up with something like make it more aesthetic, whatever that means. Right. I just know I don't like the look of it. Make it more aesthetic should do it. Let's see. It's writing code. That is pretty neat. Let's see what we get. And look, we don't even have to make cuts here. This is all happening in real time. It writes it so fast I can just watch it and talk for a few seconds and voila. I think we should be rounding it out here. There you go, the last sections. And okay, it's taking a few more seconds than I thought, but no worries, here it is. See, that wasn't too bad. And all of a sudden the website is way more aesthetic. I could have a background image here at the top. Look at that. This is starting to look good. There's little animations. I mean, are you serious? Look at these prompts. Generate a portfolio website for a designer and make it more aesthetic and you get this. Impressive to say the least. The same thing with ChatGPT would have took twice as long because there would be a bunch of copy pasting back and forth iterating here. It just happens. And as their demo videos and many people on the Internet have already shown, this is not limited to website you can build simple little games, you can create graphics, you can do all of that within this little interface yourself. You might have seen the benchmark of creating a Snake game in LLM a million times before. Now you can try it for yourself and you don't need a code editor to do it. I'm not sure if I'm getting the point across, how monumental of a development this is, because really this is a massive step from just assisting you to actually doing, AKA the difference between an assistant and something that people refer to as an agent these days. And I think that so many AI terms have a very blurry definition. I mean, how would you define AGI? You ask 10 people, you get 10 different definitions. How do you define an agent? And the worst one of them all, what the heck is even an AI expert? What I do know though is that this direction where the interface does the work for you, less and less prompt engineering, less and less copy pasting your use cases with practical results right here. This is where we are heading. Let's round this out with an example from Ethan Mollick here on Twitter where he built a game prototype that teaches about opportunity cost, but is an arcade game with Lovecraftian elements. And then all he does is follow up with make it better. Again, super generic, but these models are becoming better and better at understanding what you mean without extensive prompt engineering. Actually just created a video on why you might not need prompt engineering at all and what situations you do. Long story short, if you're doing it yourself, you might not need it, you just need basic LLM education. But if you're delegating it, if you're creating chatbots, if you're creating automations, or if you're going to be using that prompt regularly, in that case, prompt engineering is still a must. We'll be uploading that video next week, so keep your eyes out for that. But back to this little demo. It generates all the code and you have a little interactive game. And now you can build interactive games for coworkers, for your children, or just for fun. This is not exclusive to people who are who know how to write and execute code. It's not just about the writing. The writing could have been supplemented or replaced with something like GPT 3.5 already. But a lot of people got stuck on the point that they had to install VS code and they had to install Python and then maybe they had some package conflicts and they just didn't know how to resolve it. This is very common. A lot of people make it sound very easy, but what if things go wrong? You'll get stuck. Not with this. As you can see, I'm absolutely loving this feature. I think this is going to be available in every single LLM soon. And there you have it. Claude 3.5 Sonnet, freely available. One more thing, if you're interacting with it down here, you can switch to models, but there's no real reason to use Opus right now, although it's their premium model, which then also hints at the obvious feature of Opus 3.5 coming out, which would be their big model. In other words, they're probably keeping an ace up their sleeve for OpenAI's next big move. If that's GPT5 or whatever you want to name it, we don't know. But that's on its way with Llama 400B that is in training, which is going to be the open source version of this. This space moves so fast and it's very refreshing to see some of these innovations arrive in my browser usable today. All right, and if you're interested in learning about AI tools that are usable like this, every single week, I run a show on this YouTube channel called AI News you can use. Every single Friday, I gather all the updates, all the usable apps and present them to you. Plus we're doing more and more testing in the process, so it's often not just a first look, but also second or third look. Here's the playlist. Subscribe for more content like this and I'll see you soon.