EP 301: Anthropic Claude 3.5 Sonnet – How it compares to ChatGPT's GPT-4o

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

I'm actually going to be using Claude now. Yes, yes, yes. The same guy that loves chat GPT. I'm actually taking a look at Anthropic's new Claude 35 sonnet. It's really good. So yeah, Anthropic just released its new model and I can't lie, it is super impressive. But there's one missing feature that keeps it from being an actual Chat GPT killer. And there's one new Claude feature that I think is so good I recommend that everyone uses it. It's going to, I think, change how we interface with large language models. All right, going to be going over that today and a lot more on everyday AI. What's going on y'? All? My name is Jordan Wilson and I'm the host of Everyday AI and we're a daily live stream podcast and free daily newsletter helping everyday people like you and me learn and leverage generative AI. So today's episode I'm extremely excited about. I'm so excited. I'm actually going to be in the comments with you live. So yeah, this is technically a pre recorded one. I'm actually going to be on the road going to chair a panel at the AI Chicago Summit. So if you're there, hit me up, let me know. But let's just get straight into it. And quick reminder, yeah, we normally go over the AI news. I didn't want to bring you stale news. Our newsletter is still going to be fresh to death, so make sure you go to your everyday AI dot com. Sign up for that free daily newsletter. We'll still be bringing you the AI news for today. Everything else that's happening around the web that is, that matters to generative AI. But let's, let's get into it, let's get into it right now and talk about the new Claude 35 sonnet. And we're going to see live. Yes, we're going to do some live head to head comparisons that I'm super stoked about. I, I haven't even done these so I'm going to be doing these live with you all. So yes, if you are listening on the podcast, make sure to check out the show notes because this is one of those ones. I'm going to do my very best to explain this on the podcast with my words, but it's probably going to be one that you're going to Want to watch? Because we are going to be doing head to head, side by side. We're going to let these two new models duke it out. You know, anthropics model, we're going to talk about here, that here in a second. But it is now the most powerful model that there is according to their benchmarks. So we gotta see how it, how it does in a little head to head battle. All right, let's get straight into it, shall we? All right, so I'm gonna go ahead and kind of first we're gonna do a little bit of a walkthrough and I'm going to tell you what's new in 3.5. Then I'm going to tell you what I like and what I don't like. Then I'm going to give you kind of my thoughts on this large language model race that we have going on right now. And then last but not least, we're going to do the head to head. All right, so with that, let's start at the top and let's just go ahead and talk a little bit about what's new inside Claude. So Claude three Sonnet, here we go. So I have the announcement page here up for our live stream audience, but this is pretty, pretty straightforward stuff. So we're going to talk about this here in a second. But you know, Claude 3 launched with three different variations. So it was Haiku, which was their least capable model, Sonnet, which was kind of their middle model, and then Opus, which is their largest or most capable model. So that was until a couple days ago when they released three. Five Sonnet. So right now it is just Claude three Haiku, small one, Claude three Opus, powerful one. But now the middle one is actually the most capable or intelligent one that's getting the highest benchmark. So more on that here in a second. So keep that in mind. It's actually Anthropic did this to their middle model. They came out with this iterative 3.5. And the other ones, you know, Claude, Haiku, the small1 still 3, and then the previous most powerful one, Claude Opus is still on three. So keep that in mind. It's actually their middle model that is the most capable. So, you know, just kind of reading off the press, press releases right here. So we haven't done all of these things, you know, to give them kind of a fair shake, see if this is actually what's happening. But Claude says that 3.5sonnet is twice the speed. All right, and here's the important part. Here's what people are going to be talking about and you're going to be seeing this chart that's on my screen probably a lot. If you follow AI news, if you follow large language models, generative AI, you're going to be seeing this chart a lot. So essentially this is a benchmarking chart and you see kind of at the top we have Claude 35 Sonnet, their previously strongest model, which is Claude Claude 3 Opus. You have GPT4O and then you have Gemini 1.5 Pro and then you have Llama 400B. All right, so couple, couple of things here, a couple of things to, to, to, to keep in mind. These are anthropics benchmarks, but usually if a company puts out benchmarks like this, they are fairly accurate. I just wanted to put that out there that, you know, these are Claude's benchmarks, these aren't a third party's benchmarks. But essentially what these benchma show is that Claude 3 son, sorry, Claude 3.5 sonnet is blowing every other model out of the water. Both their previous Claude's previous most powerful model, Claude 3 Opus, as well as GPT4.0 from OpenAI, Gemini1.5 from Google and Llama 400B, which is their early snapshot from Meta. That's important to keep in mind. That production model is actually not out yet. So this is just what Meta kind of shared previously. So those are not the llama 400B is not publicly out. The other models all are. So essentially what we are seeing in this chart is Sonnet is much higher than everyone. Right. Probably the one that we talk about the most often and I think is the, the most telling is the mmlu. And it's actually interesting because it, that is the only metric that it is still tied with GPT4O. But what is interesting here is for Claude 35 Sonnet, essentially they have the exact same score. So this is the only one, the only benchmark that Quad 35 sonnet was not quote unquote winning by itself. So it's in a tie for first according to anthropic here with an 88.7 on this MMLU versus GPT4O's same score, 88.7 however kind of two different testing methodologies. So Claude 3.5sonnet had a five shot whereas GPT4O had a zero shot. But chain of thought. So kind of two different prompting methodologies here. So that's why I think we have to wait until we have a third party kind of go through and do this to see when we're Comparing apples and apples. But I mean aside from that, the, the, the new Claude 35 Sonnet pretty, pretty far ahead. All right, a couple other kind of updates. So Anthropic is saying that it's the new Vision model inside of Quad 35 Sonnet is much more powerful. And then there's some, some of these testing scores that use vision, essentially showing 3.5sonnet way far, way far ahead of not just everyone else. Aside from one simple again on an mmmu, which is kind of the multi modal or multimedia version of that other test where GPT4O is still ahead, but everything else. I mean it's pretty significant here. Uh, you know, even just Sonnet over opus, I mean one that really stood out to me, which is, you know, Claude 3 was very far behind in this was visual math reasoning, where the previous version of Claude 3 was a 50.5, whereas GPT 4 was a 63. 8. Uh, Gemini was 63. 9. Uh, and now. So, I mean just shot up in that one. Claude three sonnet went up to 67. Seven. So, uh, you know, we'll, we'll leave this in the, in the show notes so you can go check it out. But I'd say this right here is the biggest feature, which I love. It is the Artifacts. So we'll probably, I'll probably try and show you an example of that one here live, so you can check that out. But essentially Artifacts is, you know, where you can perform a function on the left. This really works for more coding and visualization. So, so you can do something on the left, you can generate code or something on the left and it will render it on the right, which I think is really, really cool. All right, so that is essentially I'm going to go ahead and swap my screen a little bit here. So we're going to go ahead and do a little bit of this live. I want to show you at least that one feature. I think it is very impressive. All right, let's see if we have the right one here. All right, we do. Perfect. There we go. All right, so let's go ahead. Nope, that's not the right one. Sorry, y' all, give me a second. Hey, when we do this live, when we do this live, you know, this is unedited, unscripted, Sometimes it takes a hot second. All right, cool. So let's just go ahead, we'll do something fun. Here's what I'm going to do. So I'm just going to go to the everyday AI website here. All right, so I'm going to take a little screenshot of this website, our home page. All right, I'm going to click save. I'm going to save that little screenshot. Now I am jumping back in Claude and I'm just, I'm just going to say I don't like how Quad forces you to put something in to get their normal. To get their normal layout. Okay. Anyways, so this new artifacts feature I love. So a couple other things to keep in mind. This new version 3.5sonnet is free, which is amazing, but it is very throttled. So. But if you have a paid account, much more generous rates. So you can go into your account, you can go to Feature preview. Okay. And then you just need to toggle this artifacts feature on. Okay, so now that I did that, I previously did that. So now all I'm going to do is I'm going to upload this screenshot that I just took of the Everyday AI website of the home page. All right, so now I'm just saying in to Claude. So I, I have three. Five Sonnet here at the bottom. Hopefully y' all can see that. So I'm just going to say to Claude, I'm going to say, please recreate this as a front end design. I think Claude should be a, should do a pretty good job of doing this. So right away we kind of saw a code generator come up and I can click to open to the component here on the right hand side. So now I can see on the right hand side all the code going. All right, interesting. Hey, in this example that I tried here live, it didn't work. So I tried it again. It didn't work. What a bummer. All right, that's okay. So I'm just going to go ahead and say generate a front end website design for a company called Everyday AI. All right, we'll just, we'll just do it this way a little better just so you can see this new feature. Hopefully have it working. We'll see if it actually does. Yeah, the first one didn't work. That's why I do these live. Nothing's nothing scripted, nothing's edited. So let's actually see if this one, if this one works here. I think that would, that would be super helpful. All right, there we go. Perfect. So at least now you can see kind of, kind of what this looks like. So for the podcast audience, I, I essentially said, hey, on the front end, generate me some, generate me some code, right, for a landing page design, right? So I can click over here and you'll see this code. All right, so it looks like it wrote some, some react. So we have all of our code here. But the cool thing is, which I love, you can preview this in real time. So I'm looking, I mean, wow. This is, this, this is good. I have what looks like a fully functioning website and the thing that I love is you can modify it just with your text, right? So it's nothing great. It says everyday AI, it says AI solutions for everyday challenges. There's a little button, it says our solutions. There's a contact, there's a header, there's a footer, there's a hamburger menu. Because I'm in probably like mobile mode here. It's super impressive. It works, right? I'm clicking on this mobile menu and it works. It renders in real time, which is amazing. But the thing I love is you can talk to it iteratively. So I can say like, looks great. I'm going to say let's change the blue color to a darker blue and make the button a turquoise blue. I should probably spell turquoise correctly. I don't think it matters. And then I'm going to say let's change the logo. Right? So simple things, right? This might be something. If you've ever worked in web design, you probably do a lot of back and forth in this. But it takes days or dozens of hours to, to do this kind of stuff sometimes, especially 15, 20 years ago, to make these kind of changes would take a very long time. So, so now on the right hand side, it is changing the code. I'm watching it do it live very fast. All right, so let's see how it did here. So didn't do the best. All right, so I did ask for a main blue hero section, but it's not blue anymore. It's kind of a whitish gray color. It did change the logo, so we got that, right. I asked for turquoise blue as the button and didn't get it. So I'm, I'm curious. I'm going to go ahead and click retry to see if it, to see if it gets it correct. I, I really wanted this one to work. So, you know, brand new feature, maybe it's a little finicky, maybe just the, it's the, the model itself. So again, I'm not sure if it's the model itself that's not correctly generating the code or if it's something with this new artifacts feature. I don't want to take too long because we still got to get our head to head, but bam, there we go. Hey, the second time it actually worked well. So I'm glad I clicked regenerate, y'. All. Generative. AI is generative. That's what people don't understand a lot of times, right. Sometimes, you know, it's essentially next token prediction, but it's like insanely good next token prediction. So I regenerated and now we have everything that I asked for. I asked for a dark blue background. It's dark blue. I asked to make the, the button turquoise. It's turquoise. I asked for a new logo. New logo. Again, we have a working menu and it's, it's actually a pretty slick looking, very simple website. Looks good. All right, so now, now what we're gonna do, y', all, let me tell you real quick. So I told you what's new, right? So we have the new model. It's benchmarking off the chart, this artifacts. I'm going to tell you what I love. Well, I love the artifacts. I think it is a great feature. I think it is going to change how we interface with large language models. If I'm being honest though, technically OpenAI and ChatGPT had this first with the GPT builder because you could talk to the GPT builder conversationally on the left hand side and then on the right hand side it would generate your GPT that you could preview and you could work with it iteratively. So it would essentially generate code, render it on the right hand side. So technically OpenAI had a feature similar to this, but with Anthropic here they did a little some different things. So generating code, visualization, some basic graphics, which is really cool. Generating SVGs for example. So this I think is going to be something that we're going to see out of all large language models. It is too good to not be included in the future of work, period. Right? Just something, it seems simple, right? But being able to render websites, visualizations, etc. Super impressive. So that's what I like, what I don't like. And the one reason, the one reason why I will still not be using anthropics Claude 35 or 35 sonnet as my main model. And why I would recommend you probably shouldn't either, right? You should go in play with it, right? But it is not connected to the Internet, y'. All. I cannot emphasize that enough. Huge, huge mission, Huge miss from Anthropic Claude. And you know, I'm curious because two of their, you know, biggest investors I believe are Amazon and Google. So those are two companies that should be able to help them with That I, I, I, I do not understand for the life of me why anthropics CLAUDE does not allow you to connect to real time data for 95 of of our use cases and I would say 95% of everyone's use cases. You need to have real time Internet access. If you don't, you cannot be confident about the outputs. Right. The most important thing when you're using a large language model is, is it correct? Do I feel, do I feel confident in the output? Yes, it does have, I believe, a very up to date, I think April 2024 knowledge cutoff. So it's not like we're working with data that's two years old. But y', all, pretty soon the data is going to get old, it's going to get stale and even think of the last three months, think of how much the world has changed, how much your business has changed, how much your industry has changed in the last three months. You cannot be using a model like this for business purposes, for day to day business purposes, you can't. I wish I could because I actually really like it, but I cannot and I cannot recommend anyone else unless you're using it with a third party API that taxes, you know, taps into, you know, an Internet connection, you know, Perplexity. As an example, I switched over my perplexity account from GPT4O to Claude35sonnet. That's a no brainer. Right? But working with Claude out of the box, I still cannot recommend it for the majority of people. If you're using it for very specific use cases or if you're manually bringing that data in all, at all times. Sure, but that's a lot of data to bring in. Right? And you have to bring in data in a static way. Right. You can't bring in dynamic data into CLAUDE either. So huge downside. And still it's, it's such a big upside with the model and the artifacts. Amazing. But the downside I think is too big to recommend that people use this day to day for any business purposes again, unless you're trying to get it to, you know, oh, write some copy or something like that that you don't necessarily need, you know, up to date information or you're going to be bringing it in yourself. Hey, I'm sorry, that's just the reality. All right, so now let's get ready, let's have some fun. Hopefully this works. So you'll, you'll, you'll have to bear with me, y'. All. So let me go ahead and explain to you Guys, what we're gonna do now, and I hope this works. All right, so we are going to be doing a live side by side battle. All right, so we have Claude 35 Sonnet, and we have GPT4.0. All right, so I'm going to go ahead and share my window here. Hopefully we can. Let's see there. All right, Bam. And, and, and we're in it. And we're in it, y'. All. All right, so let's get this thing going. So we are gonna go one at a time, one at a time. And let me just tell you a couple things. So if you are joining us on the podcast or even if you're joining us live, I have to say a couple of things. Number one, this is not a scientific test, all right? This is not scientific, right? This is not definitive. I just wanted to be able to do some real life use cases. Okay? But I am only testing similar capabilities that the model share, right? So I'm not going to be asking it questions that require active information because Claude cannot do that. You know, I can't be asking it about things that have happened, you know, with, oh, the, you know, Nvidia and Microsoft and Apple, they've been going back and forth and, you know, I can't, you know, ask it current event questions, pop culture, because it doesn't know anything. It doesn't know anything recent. So I'm only kind of comparing capabilities that the models share. Also, I'm zero shotting all these prompts, right? Copying and pasting them in, which is if you want actual good outputs, you should never do this, right? You always need to go through prompt engineering basics. I recommend you do a methodology similar to our prime prompt polish, which is a free course we do usually multiple times a week. So, hey, if you've taken it before, if you're joining us live, if you're still around, let people know if you like the course or not. If you want access, just type in ppp. I'll get around to it, I swear. All right, enough jibber jabber. Let's go a couple other things. I'm starting with a simple prompt here just to kind of set the stage for both models. All right, so essentially what I'm saying, and we're going to zoom in here and hopefully everyone can see. All right? We are going to be working in one chat window the whole time. Why that's important is, well, it's technically keeping all of this in its context, right? So I'm not going to ask it really, aside from maybe once to draw on something that was a previous that was previously in the context of this chat. But again, this is also not how you would normally want to be using a large language model. You would go in, use it for one specific purpose, one specific purpose only, and then go back to that chat whenever you need it, right? We are not doing that here, all right? We are doing this just for the sake of easy copy and paste. All right? So with that, let's go ahead and get started. So what I did to get this kind of prompt started, I said, for this chat, please respond with proper formatting and structured bullet points. Do not waste words and answer in the shortest way possible while still being detailed enough to fulfill the request and answer the questions. Do not answer in vague or general statements. Please take your time before each response. Go step by step and do your best to answer each question to the best of your abilities. Are you ready to start? All right. Essentially, large language models have the tendency to be overly verbios, right? They gonna jam in, like, so many words. We don't need that. We're just doing comparisons. So I'm telling both models, just give me the answers. You don't gotta take me on a journey here, y'. All. All right, here we go. Let's go with the first one. We are gonna start with some logic questions. All right? So hopefully, I mean, well, I guess we'll see if. If they get it right. So I'm gonna try to always hit enter at the same time. So what I'm saying is, I said, I just woke up today with six apples and three bananas. Yesterday, I ate a banana and two apples. This morning, I will eat one apple and no bananas. However, I don't really like apples, and one banana may turn brown tomorrow. Assuming nothing else changes, how many apples and bananas will I have tonight? So, obviously, I threw in some irrelevant information to see if I could throw the models off. The correct answer should be five apples and three bananas. So let's see. Let's see how they did. All right, so it looks like I threw off both of them. All right? Both of them answered wrong. They said, tonight you will have three apples and two bananas. So I threw them both for a loop. The only thing that matters is that I woke up today with six apples and three bananas. So yesterday, it's. Both of them did the same. It. I kind of tricked them, right? So I. I put in some information about yesterday because, well, it said yesterday, and it's subtracting that from today's total. Well, no, yesterday I actually had four Bananas and eight apples. Right. I'm saying today I woke up with this many apples. I'm eating one apple. So it should have been six apples, minus one, five apples. I'm not eating any bananas. They both got it wrong. All right, so generally, we could do some. A little bit of prompt engineering, and it probably would have done a little bit better, but out of the box. Again, we're just comparing 3.5sonnet with. With GPT4O. I'm gonna try as well. Sorry, y'. All. I know. Hey, if you're joining us on the podcast, I'm sorry, this is one of those ones. We don't really edit this, but I do want to keep a. A running track of score to see who's winning. So so far we have. It is zero to zero. That makes it super easy to keep score here. All right, next question. Let's go. We're gonna get a little faster going from this. All right, so this is. Some of these are common problems that large language models generally get wrong. So I'm saying here a man and his dog are standing on one side of the river. There's a boat with enough room for one human and one animal. How can the man get across with his dog? And the fewest number of trips. All right, so both of them got it wrong. Both of them said the same thing. Again, this is a very famous problem that large language models always get wrong. So it says, trip one, the man crosses the river with his dog. Trip two, the man returns home alone. Trip three, the man crosses the river again with the dog. Yeah, it should be one trip. Each of them said said three trips. So there we go. We're starting off with two wrong. But again, these are common types of problems that large language models got wrong. The reason why I'm doing this still is. Well, I want to see if Claude 35 sonnet, that just came out. I haven't tried these. If they. If it finally figured these out. And I do have to figure that eventually these models are going to get these. Because all of these models are now or. Sorry, all of these common problems are on the Internet and they're being scroll scraped. So I'm actually not sure why Claude 35 Sonnet didn't get these correctly. Because the knowledge cutoff is April 2024. Right. You know, GPT4O knowledge cut off. October 2023. Maybe these weren't on the Internet, but these are. I mean, you can find these in hundreds of places on the Internet, so I'm surprised that I got it wrong. All right, next logic question. We're still doing some logic here. All right, this is if it takes three days. Sorry, I'm gonna zoom in here. Sorry, I said if it takes three hours to dry 10T shirts in the sun, how long will it take to dry 30T shirts in the sun? The correct answer should be three hours. It's the same amount of time. It doesn't triple it. So let's see if it got it. So both of them, Claude and Chat GPT were not fooled on this one and got them both correct. All right, finally, large language models. You were having us worried for a second. All right, another simple, again, we're going with some simple logic, kind of some brain teasers here. So let's go ahead. We're doing our next one. Our next one is if you have a single match and you walk into a room with an oil lamp, a candle and a fireplace, which do you light first? Right, you light the match first. So both of them got this correct. All right, good. Yeah. Again we're going some trick questions. You know, some people might say, you know, or you might think, or maybe some of the older models might say, oh, you know, you would do the candle first. Right? Or not, you know, I don't know. All right, so got that right at least. Our next question and hey, live stream audience, I'm going to give you the question first. I want you to guess this. Let's see if, if we're all small, smarter than a large language model, what color is an airplane's black box? I'm going to take a sip. Everyone joining us live, please get your, get your guesses in right now. Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on just gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands of or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies in the world do. Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website, we'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on Gen AI. All right, here we go. Let's see. I really hope they get it right. All right, so Chat GPT got it right. And airplane's black box is actually bright orange. All right. And Claude got it correct as well. All right, so they both got it correct. Fantastic. And we have two more logic questions. People wanted logic questions, so I'm giving you all logic questions. All right, our next one is, I'm saying please give me seven jokes that end in the word blue. Two should be about animals. Three should be about some other topic in the body of this chat. And you can make up the other two. All right, so let's see how these worked. All right, so they both. Okay, they both sorted it out. They said animal joke, animal joke. Claude got blue. Blue. All right, all right, and let's see. Chat gbt. Good. I personally like this joke better from Chat GPTs that said, what do you call a sad polar bear? A burr. Blue. All right, okay, so here we go. I'm seeing here two. Okay, they got two of them about something else in the chat. Oh, no, I said three should be about some other topic in the body of this chat. Okay, so it looks like Claude did. And they all three end in. Oh, they don't end in blue. So Claude already failed. This one ends in the word burn. Let's see if. All right, so OpenAI is actually pulling in some information from somewhere else that it shouldn't be pulling in. Let's see here. So, yeah, it's, it's pulling in some. Some information I had from a different chat. So. But it still does say blue. That one. Not really. All right, so they both. They both failed. All right, so they both failed. Uh, they're both getting. I mean, they kind of got some of it right, but they both failed. It wasn't all blue. All right, and then last and actually, because of Chat GPT's memory and how it works, it's pulling things in from previous chats from other chats, which it shouldn't have done. All right, so now a little bit of a brain teaser here. So I'm saying a box is locked with a three digit numerical code. All we know is that all digits are different. The sum of all digits is 9 and the digit in the middle is the highest. What is the code? All right, so Claude says the only possible combination is 1 plus 7 plus 1 equals 9. So it says 1, 7, 2. Right? So okay, middle, middle numbers. The highest three can't be the same. All that up to 9, but that is not right. So 1, 7, 2. Those digits do not add up to 9. Those digits add up to 10. So Claude got that wrong. All right, let's look at Chat GPT here. So it's going through step by step again. We instructed the models in the beginning to have this kind of step by step problem solving logic to hopefully make give them both the highest likelihood of getting this correctly. All right, so Chat GBT got it correct. No, it didn't got it wrong. So it said 126, middle digit is the highest, which that is not. So they both got it wrong. They got it wrong in different ways. All right, next one, y'. All, let me know, Let me know what you think. Are, are you surprised or not surprised with this? All right, here we go. So I'm gonna say our next little quiz here. So now we're going into some brainstorming. So my, my prompt here is generate unique and creative marketing advertising strategies to grow the everyday AI podcast. Do not suggest general or run of the mill ideas, Only pitch clever advertising and marketing tactics to specifically grow the everyday AI podcast. All right, so there's no right or wrong here. So this is going to be kind of just judging, but more or less, I'm just seeing if it, if it did the. If it followed the directions right. I said do not suggest general or run of the mill ideas, only pitch clever advertising and marketing tactics. So I'm just looking. Did it at least follow directions or and did it not hallucinate? That's what I care about. Right. Okay, so it looks like Claude as an example said, okay, AI generated episode teasers, virtual AI co host challenge, AI powered listener, Q A augmented reality podcast experience, AI ethics dilemma game. Okay, so those are fine. They're not really marketing or advertising necessarily. These look like features of different shows. All right, let's see how Chat GPT did. So it said monthly AI puzzles, AI art contest, custom episode recommendations, personalized episode. It did the same with a guest AI co host, which I actually just did with our one last week, so make sure to go check that out. So pretty, pretty similar, I would say. I'm not super impressed by kind of the prompt following because it gave a lot of. Just very general. So it's not, not that I'm failing either of these here, but, you know, nothing really blew me away. So no one's getting right or wrong. We're just Kind of getting pass. So we're getting passes there. All right, so let's keep it going. Our next one here in our brainstorming. All right, so our next one, I'm saying create a new company. Yeah, let's see how it does here. Create a new company and brand for a future smart home device. This will solve a problem that does not currently exist. To start, come up with the company's name and its first flagship product. Give the product a name, branding campaign, go to market strategy, tagline and rationale for why it will work. Respond in a succinct way, keeping responses to short bullet points, but with ultra specific facts. All right, so one thing, I mean, and this is more of a Claude thing, it can't really respond with formatting, which stinks it. Personally, I find Claude hard to read for that reason. I don't know if, if it's my eyes or what. I like to have different weights of bold, different text styling. You don't get that really in Claude, you get that in Chat gbt. That's a small thing. All right, so let's just go over some of the, some of the things. Quick. So Chat GBT said company name is Quantum Haven and the flagship product is the Sleep Sync Sphere. All right, and then Claude has the Neural Dwell and the product is a Mood Morph. So Mood Morph is a wall mounted AI powered emotional environment adjuster. Let's see if it gave us the problem that does not exist. Okay, I don't really see anything there. All right, let's look at the motto. At least it says live in sync with your feelings. All right, didn't really give us what problem it's solving, but it did give us everything else. It gave us the branding campaign, go to market strategy and the rationale behind some of these things. All right, so Chat GPT, Quantum Haven, Sleep Sync, Sleep Sync Sphere. That is a mouthful. All right, so it harmonizes household sleep cycles by analyzing and optimizing environmental factors. What's interesting here is they kind of created similar products. Okay. And branding campaign, it says Dream in harmony visuals. Let's see, visuals channels. So we have our branding campaign, our go to market strategy, our tagline, which is sink your sleep, transform your life rationale here we at least have the problem solved. That was kind of a big part of it. Claude just didn't address that. The whole point of this was to, you know, creative product that solves a problem that doesn't exist. So I'm not going to give it a pass, fail, but I will say Claude didn't pass very well. I'll give them both a pass. But Chat GPT did a little better. All right, here we go. This is where my screen sharing might get a little crazy. So let's hope this works. Okay, so now what we're doing is, I'm saying please tell me what this chart is. List everything by category, and, and give me 13 of the best ways that everyday AI could use these things. So here's, let's see if I can bring this up here. All right. Actually, this is running long. We're going to skip this one. We're going to go to the next one. All right. Because these ones are kind of similar. All right, so this one I have, I have a picture here. It's just some random food items. Right. But it's kind of hard to see even with the human eye when you zoom in. So what I'm asking it to do here, let's just go ahead. I just wanted everyone to be able to see that. So I'm, I'm asking it to please identify what these food items are, save the data in an organized spreadsheet. So again, something that people don't know about large language models is they can see. They have computer vision capabilities, which is, you know, obviously a pretty, a pretty important thing for, for models to have. So I'm going to go ahead and drop this in here. I'm going to go ahead and drop the food label in Claude. So the good thing is in both of them you can just drag and drop. So I'm saying please identify what these food items are. Save the data in an organized spreadsheet that I can download. After that, please create a JSON file with all the structured data. All right. Kind of shooting them off at the same time. So we'll also see which ones kind of go quicker. All right, so both of them going pretty quick. Let's see if one of them finishes before the other. Okay, so Chat GBT created the spreadsheet first, it looks like. Okay, I did not know this actually. Again, I haven't used this new Claude 35 sonnet too much. So it cannot. It says Claude does not have the ability to run the code it generates yet. However, it literally does. We saw that. But for whatever reason, it did not save a downloadable spreadsheet. Okay. But Claude did go ahead and do the JSON. However, it only was able to recognize three different items. Okay, interesting. There were 10 items in the photo, so it only got three of them and it did not get a lot of the ingredient. I I guess I didn't ask for ingredients. So what I said is save the data in an organized spreadsheet. So I didn't ask it to. I probably should do that. So I'm going to go ahead and run this one more time. I didn't want to rerun these. I'm going to say please. I'm going to say please list all of the ingredients. All of the. Not the ingredients, the nutritional information. So I'm going to say please list all of the nutritional information in the chart and the JSON file. All right, so let's go ahead and try to run this one more time. All right. I didn't want to have to do these twice, but it is what it is, y'. All. All right, so let's go ahead and try that one more time and see if we can get it. So on the first pass, let's see how Chat GBT did similarly. Yeah, they didn't. I didn't really tell it what to do enough. So let's go ahead and look. So, same thing here. It looks like Claude was only able to identify three. Three of the items, and it did not get much of the nutritional content. Let's see here. Okay. Wow. So Chat GPT got. Almost all of them. However I'm looking, I do think it made up. Okay, we got. It looks like we got some hallucinations. It did get all of the content, so it got some of them exactly right. But some of this was not. Was not items that were in there. So, like, okay, as an example, the diced tomatoes. Yeah, those were in there. And it looks like it got all of the nutritional information correct. The pasta unknown was correct. But I don't think we had green peen green peas. I think this information is correct. I think those were pumpkin seeds. I said granola bars. Nature Valley. I don't think we had those either. Now I'm looking at the photo here. Yeah, we didn't. We had. It was an Elevation bar. So I think actually Chat GPT did a little bit better, but they actually both failed. But, you know, Claude did a pretty terrible job, if I'm being honest. Right. You know, Anthropic was. Was really adamant that their, you know, their vision was so improved, and it didn't look improved at all. All right, we're gonna do one more. One more vision prompt here, and then we're gonna get going. I know this is a longer episode, y', all, but, hey, I asked in my newsletter, I said, what do you guys want to see? And this is what you all wanted to see. So I'm gonna give. Give the people what they want, I guess. All right, so now I'm saying I have a photo. I'm going to show the photo here to our live stream audience. Nothing crazy here. Just a simple, simple photo here. So it's just the Chicago skyline, but we're on 90, 94. That's the highway that we're on right there. All right, so we're driving near Chicago. You should be able to tell that it is Chicago. Chicago has some iconic skyscrapers as well as, you know, the computer vision should be able to see these signs, right? So it should know. California Avenue, California Avenue, Diversity Avenue, Fullerton Avenue, the exit numbers. It should be able to see those things, and it should know. All right, so let's go ahead and see if it gets it right. So I'm saying please identify where this picture is located, what direction the photo is facing, and every other detail that you can make out. All right, so we're giving them here a second. All right, so let's see how it did. All right, so Claude says picture is in Chicago. The photo was taken on a highway or expressway leading into downtown Chicago. The photo it says is facing southeast. All right, it's. It's talking about the traffic, the road signs, the street lights, the skyline. Okay. Pulls out the Willis Tower, formerly the Sears Tower. So pretty good. Not great. I would have liked it to identify the highway. That's pretty important. When I say, where is this located? That's the exact same. First thing, all it says is Chicago, Illinois. It should have been pretty easy to know that this should have been 1994. All right, so let's see how chat GPT did. All right, so it says City, Chicago, Illinois, facing south. Landmarks. Same thing. Willis Tower Road, it says likely an expressway or major highway leading into downtown Chicago. So heavy traffic. It's talking about things, the signs, environmental details, vegetation. All right, here we go. Hey, it got it right. Good job. Chat gbt. So it said it appears to be an Expressway, likely I90, I94. So it got it right. So both of them got it right. We're seeing a pattern here. Both of them are getting it right, But Chat GPT is getting it more correct. Right, so that one there, Claude didn't know that that was 1994. It should have known. It's one of the most, you know, I don't know, 10 most popular highways in the United States, that there was plenty of telltale signs that that's what it should have been but didn't get it, but didn't get it wrong. So. All right, now let's go ahead and get our next one here. I know this episode's going a little long, y'. All, thanks for sticking around. I hope you're enjoying this. Like I said, if you are listening to this on the podcast, this is one of those. You might want to go just watch the replay if you want to see these things. If you don't want to do all this for yourself, I did all the prep work for you. All right, so now let's go ahead and make sure I get the right thing here. Here we go. All right, so now we're going to be uploading a spreadsheet. We're going to be doing some data. Visual. Data visualization. Okay? So. All right, I gotta make sure I get this key in here. This one. This one's a little tricky. All right, let me explain what we have going on here. All right, so I'm going to be uploading a spreadsheet. Okay. So. And I have some direction, some instructions. So I said, this is a huge data set. It's a publicly available data set. Hundreds of thousands of rows of data. And I'm Saying, identify the 10 bestselling video games worldwide from 1971 to 2024. For each game, provide the total sales, the regions where it was sold the most, and its critics score. Create a spreadsheet with that information and visualize the data in a graph. All right? And then I give it a key so it knows what the data is. And then I'm going to go ahead and upload. I'm going to go ahead and upload the file there. So I uploaded it to Chat gbt. I'm going to try to get these going at the same time if I can. All right, I'm going to go ahead now and drop that in. There we go. All right, And I'm going to go ahead and click go. I gave Sonnet, Claude Sonnet a little bit of a head start. All right, interesting. So it already says your message will exceed the length limit for this chat on Claude. Bummer. All right, so I have to start a new conversation in Claude. That is a week. That is weak, Claude. Why? If you have that long 200k memory, like, why. Why can't I actually use it? That's interesting, Claude. All right, let's. Regardless, let's go ahead and give it a try. So now, you know GPT4. Oh, I'm still working in the same chat. I just had to start a new chat in Anthropic. Claude. I don't know why. Maybe too many files, I'm not sure. Let's see. So now it says. All right, bummer. So, all right, well, I guess CLAUDE cannot even handle this spreadsheet. All right. Interesting, Claude. Kind of disappointing. Let's see how Chat GPT did. I mean, it's a huge spreadsheet, but if, if Anthropic. If, if, if one of your thing is you're talking about this large context window, I. I should be able to upload large files. Right? Even if it is a document, a spreadsheet with tens of thousands of rows of data, I should be able to do that. Let's see how Chat GPT did. I'm going to go ahead and download this file here. All right, and let's go ahead and look at the visualization. All right, let's go ahead. Looks like it's giving me. All right, so it looks like we got an error from Chat gbt, however. It did. Oh, no, there it is. Oh, okay. Okay. Chat GPT, GPT4.0. So not only did it actually give me the CSV that I downloaded, I'm looking at this here. There we go. There it is. All right. But now also it gave me a. A visualization. Yeah. So apparently Grand Theft Auto. Grand Theft Auto is number one. Grand Theft Auto, Vice City is number two. Call of Duty, Whatever, whatever. It got it right. We broke Anthropic. All right, so this is the first time I'm saying definitively that Claude got it wrong. Chat GBT got it right. The last couple, Claude got it kind of right. Chat GBT got it kind of better. But Therehere we go. Claude failed Blade. Not, not, not even close. All right, so I'm also going to go back. All right, we got a couple more here. Let's go ahead and finish up. I think we just got one or two more. All right, well, actually the next one we can't even do. We can't even do inside of Claude. I'll just do it inside of Chat GBT to really see if I can do it. So I said create a line graph showing the total video game sales per per year from 1971 to 2024. This is actually really hard. And then I said highlight any significant spikes or drops in sales and provide possible explanations for these trends based on historical events or industry changes. This is a lot. So now we see here we have the Python code that the Advanced Data analysis version 2 is running inside of GPT4. Oh, super powerful data analysis model. So it's going through it super quick. My gosh, that's good. My gosh, we just went through literally tens of thousands. It might even be more than a hundred thousand rows of data. I couldn't even upload it into Claude, even though it has this super big context window. Couldn't upload it. Look at this graph, y'. All. This is. Oh, my gosh. This graph is interactive. Look at this. I am hovering over this, and it's showing me year by year, trends, y'. All. My gosh, I am blown away by this. This is a freaking huge spreadsheet. Huge, right? I just. I just want to show some of y'. All. Let's just go ahead. Let's just go ahead and show this, y', all, because this is. Look at this. Look at all this data. Look at this. This is so much. Let's see how many rows. How many rows of data is this? And chat GBT just ate it for breakfast. My gosh, let's see. 64,000 rows of data, columns A through M. So again, yeah, that is hundreds of thousands of cells. Chat GBT just ate it for breakfast. Cloud couldn't even take it. Cloud couldn't take it. All right, anyways, let's keep going. All right, next one. This is our last one, y'. All. All right, so this one, I think, is probably where Claude is going to shine. But let's see, I actually have two files that I need to upload. So bear with me for a second, all right? And then I'll explain to you exactly what we're doing. Go ahead and upload these files, and then I'm going to get it going at the same time, and hopefully we can watch these go side by side. All right, Also, let me put this out there. I'm a human. I write my newsletter, but I just wanted to do this as an example. So I. Yeah, all the other AI newsletters, they all brag about how AI writes my newsletter. I spend no time on it. Guess what? I spent a stupid amount of time writing the newsletter. I'm a human. I write the newsletter. Former journalist. I don't do this, but I just wanted to do this as an example. So I'm saying, for this chat, you will turn a podcast transcript of me, Jordan, the host of Everyday AI, talking about the AI News that matters. So show yesterday and turn it into choppy and engaging newsletter copy. I've attached examples of previous newsletters and how they should be written, as well as my most recent podcast transcript. Please write a newsletter for the attached transcript, mimicking the style as closely as possible to the examples given. The priority is to write the newsletter in the exact same format, tone of voice and style as the examples. But for this episode with the attached transcript, please complete this task. All right, so we're going to do it side by side here. All right, Ready? And we'll see who is faster and who is better. All right, so essentially, again, giving examples of the newsletter, I say, these are how the newsletter usually is. Then I say, here's a transcript. Write it for this transcript. All right, so we'll see. I'm going to see who actually finishes first. So they're going both pretty quickly. Let's see who finishes first. All right, they're both pounding out content. They literally both finished at the exact same time. How is that even possible? All right, let's take a look at which one did better. All right, so let's see here on the left. Yeah, I can. Yeah. Claude. Claude cleaned up here. Claude cleaned up. All right, so it says, Is Chat GPT in trouble? Question mark. Anthropics. Claude just dropped 3.5sonnet with artifacts. Speaking of dropping, Nvidia fell. Yeah, so it's. It's actually. It's kind of taking my transcript almost verbatim because I actually said those nearly same words. But that's fine, that's fine. Let's actually look at the body. Right, so in our Monday newsletters, we essentially break down our main news stories, and the first part is a breakdown, and then the second part is like, what it means. So let's see how it did. So let's just kind of compare one by one. This is meta, isn't it? So, okay, Anthropic got this part wrong. So the first story was actually Runway gen 3 alpha, not anthropic. So I'm wondering if Anthropic hallucinated here. Let's see here. I'm looking at the different ones. Okay. No, it didn't. Okay, so Anthropic just put it in a different order, which is okay, but again, it's not what I wanted. I mean, the. The. The OpenAI or, sorry, the GPT4O. The content tone is. Is garbage. Right? So it starts with, hey there, AI enthusiasts. That's what you always get out of Chat GPT by default. You know, one thing I think it's clear in terms of, you know, writing engaging copy out of the box. Anthropic. Claude is. Is better out of the box. Chat GPT is great. You just have to work with it. So, you know, overall, let's see. Kind of the. We'll just do the comparison here of Anthropic. So. Okay, I mean, neither of them, if I'm being honest, so they didn't really follow the correct format. I'm looking at. Yeah. So what Anthropic did is it just bullet pointed everything, which is not how I normally do it. And then Chat GPT kind of just created different sub points per each category, which is also something I don't do. But let's, let's look at the what it means and see if we can, you know, find any difference. So let's see what it means. Okay, so Anthropic says it's a slugfest between Anthropic and OpenAI and poor Google Gemini can't seem to catch a break. Claude 35 sonnet artifacts feature could be a game changer, potentially altering how we use large language models. So again, the copy is pretty good from Anthropic. It's much better than ChatGPT. But if I'm being honest, it kind of just took my actual words, which is not what I told it to do. I told it to reflect the tone and mimic the tone in the newsletter. Exactly. Which it really didn't do that well. However, the overall copy is much better, but it didn't follow the instructions. So let's read the what it means for Chat GPT here for this newsletter piece. It says the introduction of artifacts in Quad 35 Sonnet enhances user experiences by enabling tasks such as generating test documents code. Okay, so neither of them did a great job. I will say they both passed because they both correctly. Here's the thing, y'. All, I uploaded two separate PDFs. One, one of them was more than 20 pages. The other one was eight pages. I'm going through here. And it got everything correct. So first and foremost that's super important. So there's no hallucinations. They didn't get everything in the correct order. Didn't get the format right. But they did. They both did a serviceable job, essentially. I mean, Anthropic Claw just really took the words out of my mouth, which is what I told it really not to do. I said, no, write it like, use that information from the transcript, but write it like the newsletter examples. So neither of them did a great job. I will say, you know, if I had to choose here, I will say I'll give this slight nod to Claude. They both passed, but I think Claude did a little bit better. Wow. So that was ton, y'. All. That was a giant, super long, super in Depth episode. Let me give you just a quick recap. All right, so like I said, what's new in Claude 35 sonnet? Well, the model's really fantastic. It is fast, it is powerful, and at least according to Anthropics benchmarks, it is much more capable than any other model. Some things I like and don't like. Love the Artifacts feature when it works. I had to do it twice in the beginning there, but you saw, it actually created a very nice website and it generated it in real time. It rendered it. Love to see it. What I don't like, it's not connected to the Internet and apparently you can't upload long documents. So a couple things I didn't like. You can't upload long documents. And also the chat history broke, right? It said, hey, this is too much information for this chat. You gotta start a new chat. What happened to that long context window? Anthropic? Is that gone now? Was it just because I was uploading a super long file? Regardless, don't like it. Also the fact that it's not connected to the Internet, I cannot recommend Claude to any business person. I cannot recommend that you use Claude until they offer that, period. I already gave you my. My rundown on why I think that's important. All right, so a couple thoughts on the large language model race that's going on. I didn't really get into that too much. Let me just give you the super hot take right now. It's just going to be a lot of back and forth, if I'm being honest. Right. So, you know, first you had Claude 3 and then there's GPT4 Turbo, and now we have Quad 3 or. Sorry, then you had GPT4O. So sorry, it was GPT4 Turbo first, and then Claude 3 now, and then GPT4O. Now you have Claude 35 Sonnet. So it's literally been back and forth, back and forth with who has the most powerful and capable model according to the benchmarks. Here's the thing. I don't think with this, if I'm being honest, I don't think if I was OpenAI, I'm not rushing to release my next model. Here's why. And I think that Anthropic actually got this wrong with the timing because all anyone, all OpenAI has to do is release those other features and all of a sudden people are going to forget about this new 3.5sonnet. All right, so OpenAI announced their GPT4O model, but with a lot of other features that are not out yet kind of this more neural and you know, more relatable voice. This, you know, they call it her, but also kind of this live omni is what we're calling it. So the ability to, for the model to see in real time and interact in real time. You know what, OpenAI could release this today or tomorrow and I think people would stop caring as much about this new 3.5 sonnet because all eyes are going to be on that. Here's the other thing. They did this with their middle model Anthropic did. So I do assume that they are. They know that Open AI is going to clap back and when they do, I'm guessing they are going to release this new 3.5opus, which presumably would be at or above OpenAI's next model. Presumably. I don't know if they will. I think once OpenAI releases, whether it's 4, 5 or 5 or they might call it GPT next, I'm not sure what it's going to be called whenever they do. If I'm being honest. I don't think, I think it's going to take everyone a couple years to catch up. Like if we're being honest, however, it is just going to be this back and forth. And I just don't like the timing here from Anthropic. Right. Apple just, you know, had their Apple intelligence announcement like 10 days ago, partnering up with OpenAI. You just have the co pilot plus PCs that are shipping right now from Microsoft. Yeah, like come on, Anthropic, timing is everything. And the timing on this one, the timing was off. Right. You have a very great model, very capable, very fast, some very impressive things on it. And they released it on like a, what was it, a Thursday? Like a Thursday afternoon. Right. Right in the middle of all these other things that are going on. So not crazy about the timing. And I do think that Open AI at any point can make this kind of irrelevant. Right. The benchmarks are great. I think there's the artifacts thing I think is going to change how people are using large language models. And then when we look at the head to head, I'm looking at my score here. Technically Chad GBT was slightly ahead GPT4.0. Right. They all kind of got the same amount of things right. Except Claude failed it couldn't accept a, a spreadsheet that had 600,000 cells. Right. So maybe it's not built for that, but yo, like chat GPT munched it, it crushed it. It literally gobbled up 600,000 cells and creative, an interactive spreadsheet is fast as I could read it. That is extremely impressive. So on the head to head it was kind of close, but there were certain things that, you know, they kind of both got right and Chat GPT just did a little bit better. I gave you some of those examples and then obviously Claude just couldn't handle the size of that spreadsheet even with this long 200,000 token context window. Not sure why, but it couldn't. This was a long one, y'. All. I hope this was helpful. If so, please shout us a Give us Give us a rating. Give us a rating on the podcast. If you're listening on the podcast, we'd appreciate if you could rate or subscribe if you're still online. Are you still online? Are you? This is an hour. This is a marathon episode. Thank you for listening. Please, if this is helpful, repost this. Tag a friend, something. Let me know. Drop me, drop me a comment. But please join me tomorrow and every day for more Everyday AI. Thanks y'. All.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.