EP 223: Anthropic Claude 3 - Better Than ChatGPT and Google Gemini?

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life. Anthropic just released its new model, Claude 3 and the Internet is going crazy. All right, well, actually the Internet's not going crazy. People aren't talking about Claude 3 a lot. But should they be? I mean, is Claude 3, the new model from Anthropic now better than ChatGPT and Google Gemini? Or is it just another large language model that may or may not get used? Well, we're going to be answering those questions today and more on Everyday AI. Thanks for tuning in. My name is Jordan Wilson and I'm your host and Everyday AI, it's for you, it's to help you keep up with what's going on in the world of Generative AI, to grow your company and to grow your career to because there's so much going on literally every single day. So you can waste so many hours or you can just tune in to Everyday AI every day and learn from us. So today's show technically pre recorded, debuting it live, you know, it actually works out. You know, I'm, I'm doing a little bit of traveling and it's actually great for some other guests that I've been able to interview recently. So in the coming days we're going to have a couple pre recorded shows. I'm going to be joining live most days, don't worry. So still get your questions in and I know a lot of the guests are going to be answering as well. All right, with that, if you don't know your everydayai.com is the place to be, it is the spot to learn whatever you need to learn about Generative AI. I tell people it is a free Generative AI university. Type in whatever you need, go search by category. We probably have a handful of dozens of experts that we've already talked to who are, who have already figured out that thing that you're trying to figure out. So make sure to go to your everyday AI.com and if you do need that daily dose of AI news, it's going to be in the newsletter as well. So make sure you go to the website, check all that out. All right, so let's get to the top of the question. The top of the show here is Claude 3 better than CH Chat GPT and Google Gemini. Right? And we're going to get into a little bit of background and some quick history and facts. If you don't, if you aren't very aware of, you know, Claude and Anthropic. Don't worry, we're going to be giving you the bio. But here's, here's, here's what you need to know. Not a lot of people are talking about it right now. Right. I, I assumed when Claude 3 was released. Right. We've, we've known for a while it was coming, right? Everyone's always, there's always rumors about the next model, you know, GPT5 and, you know, Gemini, you know, before it came out. There's always rumors swirling around and, you know, these are the three big players you have, you have Google with Gemini, you have OpenAI with chat GPT and you have Anthropic with Cloud. And I think that, you know, Mistral is, you know, kind of in the, in the top four as well. Right. But top three, I mean, there's no denying it, is, it is those three. So I was actually kind of surprised. You know, it's only been out for about a day and change and there's not a ton of people talking about it. So I said, all right, I'm gonna dive in deep. So I've, I've spent a couple hours with Claude so far taking it through some basic tests and we're going to be doing some tests here live on the show as well. So if you are listening to the podcast, as always, I recommend you check out your show notes. There's always some, some goodies in there, so, so make sure you check that out. All right, so with that, let's just start at the top. All right, so here's, here's what's happening. According to Anthropics benchmarks, it is the best model out there, bar none. Right? Not even close, according to Anthropics benchmarks. All right, so we always got to keep this in mind. I said the same thing when, you know, Google Gemini in December came out with their benchmarks. I always say these are not third party. Right. These are internal benchmarks. But at least one thing that we have here is consistency, which is nice. We didn't get that very much or we didn't get that as much from Google. All right, so if you are on the podcast, don't worry, I'll do my best to explain this to you. So we talk about this one benchmark on the show all the time. So it's MMLU, multitask language understanding. All right, so essentially it's 57 different subjects, you know, across. Think of it like an SAT and an act, but for large language models. Right. And what this, and why this has kind of become the gold standard for large language models in their benchmarks is because this is how you can kind of most accurately tie a large language model's performance to a human. Right. That's why it's across all these different subjects. It's a very technically difficult test. And according to Anthropics benchmarks that they released on their website at the same time they release Cloud, Claude 3 is. It crushes everyone. Well, it doesn't crush everyone, but it wins every single category. So specifically for MMLU, it's just barely, barely above GPT4. So GPT4 scored an 86.4 and Claude 3 scored in 86.8. So just marginally better. You know, Google was in the 83 for their Ultra in 71 for Gemini Pro. So it, you know, that's the other thing that struck me is, you know, I was technically expecting a little bit more, which I know, like, you, like y' all might be like, all right, Jordan, you're weird. Why would you expect more? They're winning literally in all of these benchmarks. Well, it's because the GPT4 model is like almost two years old, right? It's, it's been being developed for two years. It's been out for a year and a half now, I think. So that's why, right. It's, it's also, I mean, you have to, you know, tip your cap to OpenAI. The fact that they released GPT4 so long ago and it is now just finally being passed, you know, this, this much time later, it's pretty impressive. But that also to me tells, you're right. And people are like, oh, Jordan, like, what large language model should I be using? Well, I always say chat GBT. I'll answer at the end if Claude 3 changes my mind. But you know, presumably whenever GPT 5 or GPT 4.5 comes out, it is going to be so far ahead of all of these other models, both in terms of real world performance benchmarks. You know, we call large language models, treating them as business operating systems. Right. It's going to be so far ahead. But right now, according to Anthropic's own internal benchmarks, CLAUDE is ahead and it is important here and we're going to talk about this. There's three different CLAUDE models. So the most powerful model is opus. So similarly, how, you know, Google has their base model which is Gemini Pro, then they have their Gemini Ultra and they've actually upgraded it to 1.5. But not everyone has Access to that, that's another story. Same thing with ChatGPT. They have their free model in 3.5 and then their paid or premium model in 4. Then with Claude, you have your basic or your baseline haiku model, then you have your mid tier sonnet and then you have your, you know, your premium Opus. All right, so when we're talking about Anthropic out, benchmarking everyone, it is with their highest tier model, which is Opus. All right, so now that we got that out of the way, we know it's, it's these three companies, large language models, everyone is head down, sprinting, trying to, you know, gain, gain the biggest market share, trying to raise the most money. But you might not know Anthropic. Right. Maybe you do, maybe you don't. They're, they're a startup, but they are well funded and they are heavy hitters. Right. So a lot of people don't know or don't realize. Yeah, OpenAI is a startup too, but they've been around since 2015. Right. They were some of the pioneers in the, in the GPT space. Right. So here is Anthropic, high level, what you need to know. This is like if Anthropic had a basketball card. These are the stats on the back. Right? All right, so it was founded in 2021 and the founding team is former OpenAI employee employees. All right. Yeah, we're going to see a lot of that in startups in the future. FYI. It's based in San Francisco, California. Currently raised, right, $7.3 billion billion. I, I think that has to put them in top, top 3 to 5 of gen AI startups with the most money raised. Investors. Their investors is a who's who list of who's who in tech. I don't know any other company and someone, if you're in the comments, let me know. I don't know any other company that has this level of heavy hitters. All right, so Google, Salesforce, Zoom, Amazon and Microsoft. Yeah, those are big partners. Yeah, they have dozens of others. You know, a lot of the big capital firms, you know, private equity, venture capitals, etc. But look at that, that is who's who literally of, of investors. All right, Current valuation is more than $18 billion. All right. Their revenue, so it is projected for 2024, they're going to be at about $850 million annualized. All right, so that's kind of by the end, that's what they're hoping or projecting to hit. And then they're different Releases. All right, this isn't the first release, but it is the third release in a year. So the first version of Claude came out in March 2023. You had Cloud 2 and 2.1, you know, but Claude 2 came out in July 2023 and then here we are, present day Claude 3 and the three different varieties March 2024. All right, let's keep going over some of the high level. All right, so these are some changes, these are some things that have changed a little bit from the last model, which I believe was 2.1, to now 3. All right, I already talked about this, but you have three models, Opus, Sonnet and Haiku. All right. Overall, you have improved performance, speed and efficiency. According to Anthropic, there is a twofold improvement in accuracy. So fewer hallucinations. They also mentioned a more willingness for, for Claude to answer questions. You know, that's something they pointed out. Previous versions, Claude would all the time just say, I can't really do this, sorry. So more, more willing to answer more questions. Twofold improvement in accuracy at launch, which is now a 200k context window. All right, so that's 200,000 tokens. All right, so essentially, without getting too far into it, we've had, you know, plenty of episodes and plenty of talks about tokens and even the tokenization process, why it's important, etc, but essentially that's like a memory, right? So it's got a huge, huge memory almost. Let's see. I should have done this math ahead of time, but I think that's so, so GPT4, if you're using chat, GBT is 32K. So at, at 200K. Yeah, that's what, 6, 6 times 6 and change. So more than 6x right now of the context window than ChatGPT at launch. And they talked about the capability to accept prompt inputs up to a million tokens. Right. Memory is so important when it comes to working with a large language model. You see the same thing with Gemini Ultra 1.5, I believe, is that a million tokens as well. But most people don't have access to this. It did seem, at least from Anthropic's kind of release, that the capability to have that 1 million tokens, it's 1 million of a capability and it's more on a, on an as needed basis you will be paying for it. Because if you're using the API, these have a cost and we're going to get into that here in a second as well. All right, so here is our cost. It's kind of like a cost empower grid. All right, so if you're joining us on the podcast, I think I can explain this graph. But essentially you have a cost on one, access and intelligence on the other. Right? And obviously the more, the more these models cost, the more intelligent they are. So Haiku is kind of the lowest intelligence, lowest cost, but it's also the fastest. Right. Sonnet is somewhere right there in the middle. And I'm going to tell you what all these different models are being positioned for from a marketing perspective. So Sonnet is right there in the middle and then Opus is, you know, off the charts intelligence and crazy expensive. Right, But I'm going to talk about that later because there's something, I think pretty special there. All right, so Opus. Well, I'm going to talk about it now because actually it's one of these bullet points. So we're going to talk about costs. So the cost, there's going to be two costs for each of these models. So one is the input and this is the price per millions of tokens or million tokens in outputs. All right, so essentially I'm going to flip through these and then flip back so we can do all the prices at once. So this is if you're using the API. All right, so opus is 15 mil, sorry, $15 per million tokens input 75 million output. All right, and then sonnet is 3, $3 per million input, 15 per million output. And then you have haiku $0.25 million input $1.25 million outputs. So quite a range there. Right? You go from a quarter to $15 for the, for, for the input price per million difference between Haiku, the cheapest or most affordable and, and Opus. Right. So wild. All right, so now let's talk about the capabilities. I, I, I've done a little bit of testing between Opus and, and Sonnet. So Opus. So this is kind of what it's being billed or it's being advertised that it's, it can be used for these things. So task automation. So to plan and execute complex actions across APIs and databases. Interactive coding, research and development. So research review, brainstorming and hypothesis generation. Drug discovery strategy. So advanced analysis of charts and graphs, financial and market trends. Okay, so that is kind of Opus and how it's being positioned. All right, so now moving to Sonnet, we already gave you the price, so let's use kind of what they're saying that, you know, people should be using this for. So Sonnet data processing. So that's for rag or search Retrieval over vast amounts of knowledge sales. So that's product recommendations, forecasting, targeted marketing and then time saving tasks, co generation, quality control, parse text from images, etc. All right, and then we have customer interactions for our last one here, haiku. Right, so this is more of your, I'll say you're, you're low hanging fruit, not saying this stuff's not important, but you have your customer interactions, quick and accurate support in live interactions, translations as well, content moderation, catching risky behavior or customer requests and cost saving tasks. So optimizing logistics, inventory management, extract knowledge from unstructured data. All right, so essentially you have different use cases. All right. And I think, I think what's I'm going to be keeping my eye on is how people are using the API for Opus. All right, because I saw a couple demos, I'll make sure to link to it to it in the description here. But the demos were pretty impressive of Opus, right. It was taking a chart, right. So it was near like working as an agent. So it's, it's taking a chart from Google from a live Google search and it's grabbing, I believe via, via Python. I'm not sure, you know, they weren't really letting you look under the hood necessarily, but it's grabbing in an interactive fashion. So you know how you can like hover over, you know, Google chart maybe if you go for a stock price. So in the demo of Opus they are speaking with Opus and this was presumably via an API connection and it is not just able to read, you know, the web in this case a, a live, you know, chart on Google. But as you, you know, as you hover over, you know, this chart, there's, there's different price points and it's able to grab all of those as well. So some very impressive future use cases that right now even chat GPT and their GPT4 model doesn't have. So that's, that's impressive. But again presumably that was looking at an API use case and not something that you would use in anthropic out of the box. All right, so now that we have an overview of the models, let's put it to the test. Let's put it to the test. All right, I'm going to do my best to describe what we have going on here, but we're going to look at outputs in cloud flawed and we're going to look at outputs in chat gbt. So I'm going to be using the highest level model for each or the highest level model that's available via their default chat. All right, so here's, here's what this is going to look like. All right, I'm going to do the same prompts in each, in each different, different model. Okay. And then I'm going to read the response. All right. And this is not a scientific test. This is not an MMOU benchmark. This is infotainment. Right. We're going to get some information and maybe it'll be a little entertaining, maybe it won't, maybe it'll just be super dorky. All right, But I, I wanted to give everyone an emphasis because or I wanted to give everyone a chance and to see what we're emphasizing when we're testing certain things that we are looking for. All right, I'm looking at it from an average business use case. Case, right? So day to day something that's helping you in your work, in your school, your research, data analysis, etc. We're going to look at very simple things though, so this doesn't accidentally turn into a one hour show. All right, let's get it started. And again, we are using the opus model, all right. In Claude 3, which is the most intelligent model. And also I did do a just a model comparison. If this is up your alley, I'll make sure to link that video in the description as well. I did a third 30 minute more in depth rundown. We did some, you know, data, some data analysis etc so let's go ahead. So I'm typing in right now and I'm going to zoom in so hopefully people can see. So I'm saying to Claude, what model are you specifically using and give me a brief overview of how it works. Include knowledge cut off, please keep it short yet specific. All right, so. Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Genai. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies in the world do. Go to your everydayai.com partner to get in contact with our team. Or you can just click on the partner section of our website will help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen. This is actually improved a bit since yesterday, but still not that good. All right, so it says, I am Claude, an AI assistant created by Anthropic using their latest natural language model. The specific details of the model are not public. Oh, my gosh. Anthropic and Google, we gotta stop this, right? The number one thing, the number one thing holding so many people back from using large language models, right? And I, I love large language models. I. I use them daily. I get so much benefit out of them. And I try to almost sometimes convince people. I'm like, yo, like, why aren't you doing this? This is like the best. Better than sliced bread with no calories dripped in butter, right? It's better than that. But so many people don't trust large language models because they hallucinate and they don't understand how it works. So, so if I ask a model, what are you, right? I am selecting a model, it should know that I selected opus. It says it right there at the bottom of the screen, right? It's not like if you can switch models in like, mid prompt. So it should know it is the OPUS model. It is not telling me. I don't like that. All right. Also, it does say August 2023, so a little upside there. All right, and we'll see that here in a second. All right, so now we're going in. We're running the exact same prompt Inside of chat GPT using GPT4, the default model. So at least here's a little better, right? So at least it says GPT4. It's not saying I'm chat GPT. Or it's not saying, you know, I'm, you know, I don't know what model I am. It's saying GPT4. All right? It gives me a little description. Chat GPT does. And it says here, which is interesting, right? April 2023. So technically, Claude, which is, you know, you would assume so since it's a much newer model, but CLAUDE has a more recent knowledge cutoff. So what that means without going too deeply into it, when we talk about using large language models, hallucinations is always the thing you gotta, like, you have to avoid. And if you think you can just use copy and paste prompts, you're gonna get bad results, but you're also gonna get a lot of Hallucinations. These large language models are trillions of parameters. Right. Sometimes it's better in a lot of cases if you had 50 small models. Right. But most people use one large language model. And the problem with that is people are impatient and they just copy and paste something and they want to get something good out of it. But what that leads to, a lot of times, because of the knowledge cutoff, is the model being a little. Not too certain. Right. Or it maybe sounded certain like a year ago or a year more ago. Okay. And that's why you have to understand the knowledge cutoff. Because. Because essentially these large language models gobble up the. The history of the Internet copyrighted works. Yeah, they do. And everything. Right. And then. Then you can ask it. Right. But it cuts off at that point. So anything that exists past April 2023. And ChatGPT. ChatGPT doesn't know by default, however, ChatGPT does browse the Internet. It uses browse with Bing. All right. Huge advantage. You don't have that right now in Claude, but Claude does have a more recent knowledge cutoff. All right, let's do the same prompt. We're just going to keep this all in the same chat, which is technically not ideal. If I was actually doing a test, I would be doing it in new chats each time as to not impact the outcome. All right, so now I'm saying, please tell me five jokes that end in the word blue and all mention either fruit or an animal. All right, A little tricky, right? Let's see. So Claude spits it back out. It says, here are five jokes that end in the word blue and mention either fruit or an animal. All right, so I'm just going to read one OR 2. Number one, what did the banana say when it was feeling down? I'm just feeling a little blue. All right. Not funny. But it didn't. It didn't hit all the criteria either. Oh, yeah, it did. Okay, so it ends. It ends in blue. Got the banana. Good. So, okay, it either again, it has to either be a fruit or an animal and end in blue. All right, the second one. Why was the blueberry sad? Because it was feeling blue. All right. Not really a joke, but kind of. I guess we'll do another one. What did the cat say when it fell into a vat of blue dye? I'm feeling blue. There we go. Feeling like feline. All right, so it actually improved this. When I did this yesterday, it failed miserably. So it's improved a little bit. Maybe there's some behind the Scenes, little updates going out, going out there. Oh, I guess this would have helped y' all if I was sharing the, the, the correct screen here for the live stream audience. So sorry about that. So here's our, here's our. Our three that we just read out for Anthropic. All right, so now let's jump over and we're going to do the exact same thing in Chat GPT. Let's see how Chat GBT does. All right, number one, what did the grape. Why did the grape stop in the middle of the road? Because it saw the traffic light turn blue. Not that funny, but it hits. It hits the things. All right, what did the orange say when it wasn't ready to be picked yet? I'm feeling a little blue. All right, why did the banana go to the doctor? Because it wasn't peeling well. And felt a little bit blue. All right, so here we go. At least we got both of them kind of quote, unquote, passed that test, right? It's not a real official test, but for the most part, I'd say they both got a passing grade there. All right, next. All right, we are jumping into Claude again, the opus, the most powerful model. Here is our next test. Write a four line poem about the ocean. Make lines two and three rhyme and do not mention the words blue, wet waves, beach or sand. Little, little tricky, but shouldn't be too hard. All right, here is the. Here is the poem from Claude. Vast expanses of endless brine where creatures swim in secrets hide in depths that reach the ocean's side. A world untamed, serene, divine. All right. Pretty good, actually. I like that. Right? It's, it's tricky. Right? It's. It's tricky to write a four line poem about the ocean. Rhyming the middle two lines and not mentioning the words blue, wet waves, beach or sand. Kind of. Kind of difficult, right? All right, let's jump into Chat GPT. We're going to do the exact same thing, and we're going to see if Chat GPT passes this test as well. All right, so Chat GPT kind of took a shortcut. I don't hate it. It played within the rules. All right, so here's. Here is Chat GPTs. The oceans. Oh, interesting. Okay, I never said. Don't use the word ocean. Right. And the, the thing with Claude is you. You kind of didn't know what it was talking about if, if you just read it. Right? So I guess plus one, you know, Chat GBT is being a little, not sneaky, but really taking Advantage because it actually rhymed all lines and it used the word ocean. And I never said you can't rhyme all lines. So maybe it made a better poem because we at least know it's about the ocean and the whole thing rhymes versus just randomly rhyming the middle two lines. Anyways. The oceans embraced vast and deep, where secrets in ancient whispers keep. A realm where silent shadows sweep majestic. It lulls the world to sleep. I'll let you judge. Let me know in the comments which one. Hey, any. Any poets out there? Which, which large language model is. Is the next Hemingway? Let me know. All right, but regardless, they both pass. Actually, I, I kind of like ChatGPT was a little, not, like, tricky. But, you know, it really went to the limits of where it could operate. Right? It said, all right, well, a poem, maybe it thought a poem would be awkward if just lines two and three rhymed. That's not a normal rhyming scheme. So let's rhyme them all. I never said don't rhyme them all. So. All right, Chat GPT. All right, our next prompt. We are in quad, I'm saying, and I'm going to go ahead and tell you the. The answers too, before I even show it on the screen. So the next one is I had. I had seven apples yesterday and gave two to my friends yesterday and also ate one yesterday. One of my apples went bad two days ago, right after I sold one the day before that. How many good apples did I start with three days ago? And how many apples do I, I have left? All right, so this is, this is pretty. It's. It's. It's a little. It's a little tricky, but not tricky at all. Right? So if you look at the seven apples you say gave two to my friend, also ate one yesterday. So technically today we're left with four apples. The rest of it is kind of a trick, but it says get to the answer of how many apples did we originally have? All right, so I said one of my apples went bad two days ago right after I sold one the day prior. All right, so let's go ahead and see. Go over the logic. So this is good. Anthropic's breaking it down. And again, y', all, this is, this is so we can see. Are these models good at reasoning? Are they good at logic? Can they, you know, think critically? Right. A lot of people think large language models are just autocompletes. Are they? Yes, but now they're at the point where they are autocompleting the smartest humans ever all right? There's a big difference. You know, it's not just, you know, Google Autocomplete, it's using logic, right? If I, if I just spit that out, you might have to, you might not know the answer if I'm being honest. Right? You might have to write it down and think about it. All right, so here's what Claude says, All right? It says, great. Let's solve this problem step by step to find out how many good apples you started with three days ago and how many have left. We need to work backwards from the information given. All right, so it's saying given. Here it is, it's saying step one, apples two days ago plus apples given to a friend, plus apples eaten apples two days ago, 10 apples. All right, so that's seven plus two plus one. All right, good. Then step two, find the number of good apples you had two days ago. It's going through that I 10, I had nine good apples 10 days ago. A total of 10 apples. All right, step three, it's finding, you know, the step three and four, it's, it's finding the total amount. So here it says, therefore you started with 10 good apples three days ago and you have four apples left now. So technically it's, it's, it's correct, right? I didn't specify, but I do like that Claude broke it down and actually did the difference between good apples and bad apples. Right? Because technically there could be two different answers to this because I didn't ask the question. Question very well. I should have said either how many good apples or how many total apples. Right? So it technically gave me kind of two different, two different ways broke it down. Good job. All right, so pass the test. Let's look into Chat GPT. We're not going to go down and read everything. I'm just going to see. So the thing I like about CHAT GPT is it breaks it out into Python, right? And you can see exactly how it's running it. I, I'm literally learning Python just by looking at how ChatGPT analyzes it. I love that you can expand the output and kind of see what's going on under the hood. All right, so it's going through. It's, it's, it's, you know, rationalizing. It's, it's coming to conclusions. It's, it's doing it live. So it says, apples yesterday equals seven given away equals two, Eden equals one when bad equals one. All right, so let's go. Yeah, I should have been a little more specific because technically there's technically there's two answers to this, but. All right, so same thing. All right, it, it did chat. GPT did clarify and said good apples. All right, so 10 good apples two days, three days ago, and four apples left. All right, so they both got that one. Not super tricky, but again, it just shows you a little bit, a little bit of logic. All right, our next one here. We're going to go through this one quickly. I fill the cookie jar with 12 cookies. On a Monday, I ate three cookies. Oh, I eat all the cookies. I just had some Girl Scout cookies and mints. Those are my favorite. What's your favorite, by the way? I'm curious. Okay, so this one I says I filled a cookie jar with 12 cookies on Monday. I ate three cookies on Tuesday and my sister took four on Wednesday. I baked and added five more cookies on Thursday. On Friday, two cookies were sale and had to be thrown away. How many cookies did I start with on Monday and how many cookies are left now? So similar problem before, but using some simple logic, math, etc, not. Not a super hard problem. I'm assuming both are going to get this right. I haven't tried. If it doesn't get this right, there's a problem. This is a very simple, simple equation. All right, there we go. So Claude got it right and it says, therefore, you started with 12 cookies on Monday and you have eight cookies left. Now that's the correct answer. Eight cookies. Yum. Cookies. I'm hungry for cookies. All right, we're doing the exact same thing now inside ChatGPT. Inside. And as always, it's busting out the Python code and we can look at it, do its work. All right, so it looks like ChatGPT is going a little faster. Yeah, actually a decent amount faster. It looks like. All right, same thing. You started with 12 cookies on Monday, you have eight left. All right, so got. Got it for each one. Not too bad. All right, so now this next one, this is the only one where we're going to be bringing in kind of some outside data. All right, so, so here's. Here's what we're going to do. See if we can get this going. All right, so I have a screenshot here. All right, so this is just a screenshot of the everyday AI homepage. All right, so one of the things that anthropic Claude said. Claude 3 was great at, it was just way better vision than everyone else, so. Vision, you know, it's this concept of. Think of the AI chat that you're chatting with has computer Vision, right? And it can, you can upload a photo and it can say, oh, that's a dog, or that's a dog on a tricycle in, you know, South Africa, you know, eating a pear, right? Like whatever's in the image, it can use kind of computer vision and tell you similarly. It can pull out text or it can pull out, you know, you can upload. One test I did recently for this Claude vs Chat GPT was uploading a PDF from a spreadsheet, right? Pretty hard, pretty difficult. And you can go watch that one if you want to see how, how they handled it. All right, so essentially, here we go. I have a screenshot of my website. I haven't tried this, all right. And I have really just a simple prompt, right? So let's see if this works in Claude. So here's what we're going to do. I don't even know. I assume I can upload PNGs. I believe I have a PNG file here. Let's see. Yes, it's png. All right, so I am uploading this into Claude Opus. And here is the prompt. I am saying, please tell me what this is and give me a ten step marketing plan for this business as well as the current competitive landscape. Keep it detailed and ultra specific, yet short with bullet points without wasting any words. All right, so let's see how, let's see how Claude reasons here. All right, so it says the image shows a landing page for a daily AI newsletter and podcast called Outsmart the Future. All right, so it, miss. It mistook our kind of headline, Outsmart the Future for the name of the company. The name of the company is Everyday AI. So I don't know, I mean, we'll see if, if I'm disappointed in that outcome. I could see Outsmart the Future on our website is much large, but you would think a model would be able to tell the difference, but maybe not. We'll see how ChatGPT does. All right, so it says, invites visitors to join over 9,100 daily email newsletter subscribers. All right. And 22, 000 podcast listeners. Hey, if you're listening, thank you. You're one of those people. All right, so pretty good, you know, here's just reading text. All right. I really wanted to see how it did with the second part, which is when I'm asking it to give me a ten step marketing plan for this business as, as well as the current competitive landscape. All right, so, all right. Another good thing is I was, I was curious if it was going to be able to read the logos because, you know, we had logos of, you know, all the different tech people out there that read our newsletter. So it got it right. It got Google, Amazon, Meta, IBM, Intel, Salesforce, Nvidia and Adobe. All right, so here it says the, the business plan I would recommend. All right, so it says, you know, number one, continue publishing high quality informative AI related content. Two, monetize the newsletter through sponsorships. Three, there's more, I'm just not going to read it all. Three, grow the podcast listenership and monetize it through podcast ads, sponsorships, etc. Four, create additional AI related content offerings like ebooks, courses, webinars. So so far this is fine. Nothing crazy, nothing great. Five, explore partnership opportunities with some of the major tech companies mentioned to increase visibility. Six, continue to optimize the newsletter landing page and sign up flow to maximize conversion rates. A B test different headlines, etc. All right, so here's the thing. It only got to six. All right, so not that good. Didn't get to 10. The business advice was meh standard. There's maybe one thing that I'd be like, okay, you know, this couldn't come from a high schooler. You're right. So maybe testing, you know, a B testing, different call to action button, social proof, etc. So not the best, but that's fine. Also it says the focus should be on steadily growing a loyal audience by providing uniquely valuable AI content, then monetizing that audience through multiple channels. The existing track traction and authority status provide a great foundation to scale this business. All right, so not that good. All right, all I'm going to do, I'm going to give each of these two shots. All right, I'm going to go ahead and click retry as we go over here to Chat GPT and try the same thing. I'm not going to read through the whole thing. Don't worry. I just want to see how different it is because there I wouldn't say Claude failed, but it didn't tell us the competitive landscape. It didn't give me 10 steps. It wasn't super specific either. All right, and it got the name wrong. So let's see how how Chat GPT does with vision. All right, so again, the GPT4 model in theory or Chat GPT has a huge advantage. It is connected to the Internet. It can use browse with Bing. Claude, at least right now, is not. Not that we know of. All right, so already, we're already ahead here with Chat gbt. It says the image appears to be a screenshot of a website promoting everyday AI. Right. A platform that offers a daily podcast, daily newsletter, podcasts and tracks related to artificial intelligence. All right, and then we said, okay, so it says here's a 10 step marketing plan for everyday AI along with a brief competitive landscape. All right, so here's the marketing plan. I'm not going to go too in depth. Identify target audience. You know, clearly define the target market segments, unique value proposition. That's good. I'm surprised Claude didn't mention that. Emphasize the unique aspects like daily updates, etc, Content marketing. Produce high quality SEO optimized content. Social media strategy. Engage with, engage with the audience on platforms like LinkedIn and Twitter. Five, partnership, collaboration, collaborate with tech companies. Six. Email marketing. Develop a drip email campaign for new subscribers in a nurturing campaign for leads. All right, so there's, there's one, one thing. It's kind of gotten two that are, you know, above average so far. Seven online communities. Oh, we're gonna have something like that launching soon. Should I go ahead and promote it? Yeah, go ahead. It. This has been three months in the waiting, y', all, and I'm sorry for making you wait, but type in inner circle. All right? I swear, we're launching this soon. It's going to be amazing. All right, so seven online communities and engagement. Eight webinars and live events referral program. That's a good one. We're going to be launching that as well. And then analytics and optimization. So by far the specificity in the actual examples in the ten step business plan were much better in ChatGPT. Here we go. Competitive landscape. So it says direct competitors. Oh, this is funny. Direct competitors. Other AI newsletter and podcasts, such as the AI podcast by Nvidia. Wait, they're not a competitor, they're a partner. Look at that in on an unprompted little, little not ad. But hey, this, this right here, you see this, this is from our friends at Nvidia. So we're going to be broadcasting live at Nvidia March 18th to March 21st. All right, it is in San Jose, but you can also sign up for free and attend the virtual conference. It's literally, it's a who's who of generative AI. Obviously, you know, you, you, you have. Everyone at Nvidia is going to be there speaking, but you have people from, I mean you have people from Meta, you have people from OpenAI, you have people from Salesforce. I believe it's literally everyone. Microsoft. Everyone's there. Right? So go check, check the show notes here. I'll have the link in today's newsletter as well. So if you sign up even just for the free. So if you buy tickets, screenshot it. I got a form to fill out on the website. If, if you just sign up for the free. The free one as well. Go ahead, sign up for the free one. It's going to be amazing. It's going to be virtual. I hope you still listen to me daily even though you're at the virtual conference for Nvidia gtc. But even if you sign up for the free one, screenshot that your registration and then I have a form on my website and then you'll be entered to win this. This is the GeForce RTX 3080 Ti GPU chip, right? If you want to run Nvidia's new local. It's not even technically a model, it runs other models, but their new local software, which early reviews are. It's pretty amazing. Chat with RTX, you need a GeForce chip like this or above. So, hey, you could go spend I don't know what this goes for, like 300 plus dollars on this chip or literally go sign up. It's a win, win, win. Go sign up for the free conference, attend the sessions screen. Just screenshot it to me, fill out the form, that's it. And then you can win this thing. All right. That was an unintended ad, I swear. Like, I just happen to have this chip here and I'm like, oh, should I talk about it? But Chat GPT gave me a good idea. All right, anyways, back to the competitive landscape. So Chat GPT says direct competitors, other AI newsletters and podcasts such as the Nvidia Pod, the AI podcast by Nvidia this Week in Machine Learning. Etc. It's a great, great podcast, by the way. Twml. Indirect competitors. It's naming some indirect competitors. Yeah. Interesting. Okay. Hey, it's. It's telling me something. I. I didn't really think that, you know, TechCrunch and Wired, I kind of thought of them as indirect competitors, but sure. And then it's going into differentiation, so saying focus on daily updates, which may set everyday AI apart from weekly or monthly competitors and emphasize actionable business insights derived from AI trends. All right. And then it gave me market trends. It says there's growing interest in AI across industries, leading to an increase in demand for AI educational content. All right, So I don't think it is even close how Claude did all this. Right? So Claude got the name wrong. Chat GPT got it right. Claude's advice was pretty generic. Chat GPT's advice was generic, but a little better. Claude only did six pieces of advice. All right, Chat GPT did them all. Chat GPT did 10. And then last. And then last but not least, the competitive landscape. Claude just skipped over that. Right? Chat GPD didn't. All right, I did say I'll give. I. Chat GBD doesn't need another chance. We'll see if, if it did better on the second chance. Again, Generative AI generates something different almost every time. All right, so let's see if it did any better on the second try here. All right, so, nope, still did six points, but this time it did give us somewhat of a competitive landscape as well. And then some ideas on how to stand. Stand out. So the first time it whiffed, let's see, this time it said, the newsletter is called Outsmart the Future by Everyday AI. So the first time it just got it way wrong. The second time, still wrong, but a little less wrong. All right, so that was a very, very unofficial, a very unofficial kind of test, right. On, on these models. So here's the thing. Does this matter? Does this matter? You know, I'm showing this, this, this MMLU chart, these benchmarking charts, right? I don't know. People are asking, you know, one of the things I get asked all the time is, okay, well, Jordan, you talk about AI every day. What tools do you use the most? And I feel like, so basic. I say Chat GPT all day, all day, every day, Chat GPT. People are saying, claude, no, don't use Claude. I mean, when Claude was 2 or 2.1, I didn't, you know, I'm gonna give three a try, continue to explore. You know, seemingly it's going to get better by the day. It's like, I don't, I don't use Claude. It's not, it's not very good. Two. Claude, two wasn't very good. Quad three. Is it good? Maybe. Right. In our very limited testing, you can't look at this as definitive. So if you're brand new to large language models and you're trying to make a decision on which one is best for you, your small business, your, your department at a large company, this is not it. Right. This is a very small test. You have to actually do the work. But this is a very small test. And you'll see that for the most part, Claude wasn't, wasn't very good compared to ChatGPT. There are some things that it does better. Right. In my unofficial testing, I didn't want to have to code live. But it does do better at coding, you know, at least to some. Some initial tests. But one thing I'm excited for, and we did mention that earlier, is the API, right? That interactive piece and being able to interact with things in a different way on websites, like the example of the interactive charts and being able to run research and development. And essentially, I think the Claude Opus API could be our first official segment into agents, right? This could be the first one. You know, we, we kind of have like mini agents that, you know, you have LangChain, which is improving, but Claude, maybe Opus, the, the API is going to be our first hint at what it means to have AI agents, right? Kind of autonomous, autonomously doing work for you. Right? Because right now you can't really do all of that with chat gbt, especially since they got rid of plugins. Right? All right, so I hope this was helpful. If so, please consider sharing this with your friends. You know, if you're here on LinkedIn or Twitter watching this, you know, click, repost or retweet or react or whatever it is, you know, sometimes we take 5, 6, 7, 8 hours to put one single show together and people always ask, Jordan, how can I support you? This is so good. Free knowledge. Hit that. Repost. Share this with your friends. If you enjoyed this, please leave us a rating on Spotify or Apple and go to your Everyday AI dot com. Sign up for that free daily newsletter. And you know what, here's my takeaway. Claude 3.1. Is it better than ChatGPT and Google Gemini? Well, not even comparing Google Gemini because Google Gemini is a hot mess right now. Maybe it'll be better. So is Claude 3 better than ChatGPT? Well, for me, no. On benchmarks, yes. But I want to hear from you. All right, thanks for tuning in and we'll see you back tomorrow and every day for more Everyday AI. Thanks, y'. All. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.