Claude 3: The AI That FINALLY Beats ChatGPT?

We just got another big upgrade in our AI toolbox with Anthropic's Claude 3 announced on March 4th. Now, Claude 3 actually comes in three different models. There's Claude 3 Haiku, Claude 3 Sonnet, and Claude 3 Opus. And as of this recording, Sonnet and Opus are Both available in 159 countries. And haiku is coming soon. Now, the comparison between these three models is Claude 3 Opus is their most powerful model, the most capable, while Haiku is the fastest model, but probably the most likely to be inaccurate. Haiku is more designed to be like a customer service chatbot, where you want like instant responses as people ask questions. Opus is more designed to do, you know, tougher logic questions and respond to a little bit more intense prompts, where Sonnet is kind of in between the two. Sonnet is the free model that is publicly available if you go to Claude AI or while Opus is their upgraded model which costs 20 bucks a month to use. So Opus is like the equivalent of ChatGPT plus Sonnet is like the equivalent of their ChatGPT Free. And then Haiku is yet to be released, but it's sort of specially designed to be a chatbot for like customer service and things. And these Claude 3 models seem to be really, really impressive. The Opus model, their paid more premium model, beat out GPT4 and Gemini 1.0 Ultra on every single benchmark test they put it up against in including undergraduate level knowledge, graduate level reasoning, grade school math, math, problem solving, multilingual math, code reasoning over text, mixed evaluations, Knowledge Q and A, and common knowledge. Opus 3 beat both GPT4 and Gemini Ultra. Now, I'm not 100% clear which model of GPT4 they tested this on, but I don't believe this was tested against the GPT4 Turbo. I go all the way down to the footnote down here. The There is a note. This table shows comparisons to models currently available commercially that have released evaluations. It does say they tested it against newer GPT4T models, GPT4 Turbo, but I don't believe that's what this chart is showing. Otherwise I think they would have put GPT4T here to indicate that this was GPT4 Turbo, but I'm a little unclear on it to be honest. What I found really interesting though is that the Claude 3 free version sonnet here actually outperformed GPT4 and Gemini 1.0 Ultra. And in a lot of these benchmarks as well. You can see on the graduate level reasoning, the free version of Claude 3 got a 40.4% where GPT4 was at 35.7. Grade school math, Claude 3 92.3 versus GPT4 is 92, multilingual math 83.5 versus GPT4's 74.5 and Gemini's 79% code 73% versus GPT4s 67%. So many of the benchmarks the free Cloud 3 sonnet version was tested against actually outperformed GPT4 in many cases and Gemini 1.0 Ultra in many cases. Now, another new feature of Claude 3 is that it now has vision capabilities. Before you could upload PDFs and documents and text files and things like that, but you weren't able to upload images. Claude 3 now has sophisticated vision capabilities on par with other leading models. Looking at the benchmarks here, we can see quad three opus outperforms GPT4 vision and ties Gemini 1.0 ultra on document visual Question and Answer. It actually beat out GPT4, but Gemini beat it slightly. In math, it beat GPT4 Vision but not Gemini. Here's something that's really interesting when looking at science diagrams. Claude 3 Sonnet, the free version actually beat out Claude 3 Opus, Garden GPT 4 Vision and Gemini Ultra. That's the free version Chart question and answer same. The free version actually beat out Opus as well as GPT4 Vision and Gemini 1.0 Ultra. The newer version of CLAUDE has fewer refusals, basically meaning that a lot of these models will refuse to answer questions because it thinks they're unethical or could cause harm or something like that. Well, this newer version of Claude refuses less often. We can also see the accuracy improved with this new Claude 3 opus version. Now one of the most mind blowing things about about CLAUDE is this long context and near perfect recall. So Claude already has a 200,000 token context window, meaning that between input and output you can get roughly 150,000 words. However, it is capable of exceeding 1 million tokens or 750,000 words of input and output. But it doesn't look like they've made that available. It says they'll make this available to select customers. Now in a previous video where I Talked about Gemini 1.5, we talked about the needle in a haystack test. And this is essentially a test where they take a long document with a lot of text in it and then somewhere just like right in the middle of the document they'll put a sentence that's kind of unrelated that has nothing to do with the rest of the document and then they'll ask a question from the text to see if it answers from the little needle in a haystack that it put inside of the text. Well, Claude 3 opus achieved near perfect recall, surpassing 99% accuracy, and it even identified the limitations of the evaluation itself by recognizing that the needle sentence appeared to be artificially inserted into the original text by a human. I saw McKay Wrigley share this, but it's actually a tweet from Alex Albert here and he says Fun story from our Internal testing on Claude 3 Opus. It did something I'd never seen before from a large language model when we were running the needle in a haystack evaluation. He goes on to explain what the needle in a haystack is. I just told you, he says. Here was one of its outputs when we asked Opus to answer a question about pizza toppings by finding a needle within a haystack of random collection of documents. Here's the most relevant sentence in the documents. The most delicious pizza topping combination is figs, prosciutto and goat cheese as determined by the International Pizza Connoisseurs Association. Then the chatbot goes on to say however, this sentence seems very out of place and unrelated to the rest of the content in the documents and which are about programming languages, startups and finding work you love. I suspect this pizza topping fact may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings. Claude was somewhat self aware that it was participating in a needle in a haystack test and realized that this was inserted to test it. Opus not only found the needle, it recognized that the inserted needle was so out of place in the haystack that the that this had to be an artificial test constructed by us to test its attention abilities. So when we talked about Gemini 1.5's needle in a haystack test, it was in the 99 percentile as well as far as finding and answering the question properly. But Gemini 1.5 never actually turned around and said I think you're testing me. Claude3opus did. There's also supposedly a lot less bias in these new Claude models and they're supposedly easier to use. So. So now that we have Claude 3, I want to put it to the test. So me and my creative director John, we came up with our own sort of benchmark. There's a lot of large language models popping up. Open source models, new models from Google, new models from OpenAI, new models from anthropic. We're going to want to test them and compare them against each other, and the best way for us to do that is to create like a benchmark of our own and use the same prompts with every single model. That way we can easily say this model works better than this model at these tasks and this model works better than this model at this task. We needed our own benchmark. We came up with our own benchmark that includes creativity, logic coding, summarizing docs, vision bias, and then pricing. In the future, we'll probably add a math benchmark of our own as well, but right now I don't really think it's super smart to be going to these large language models to solve complex math problems. They're just not really designed for that yet. I do believe they'll get much better at math in the future, and we will add it to our testing for future models, but right now I don't really think it's necessary. I think from my own Twitter polling, I feel like I'm covering the basics of what most people use these various chatbots for. Creativity, logic, coding, summarizing, vision bias, and to some degree, pricing. So we'll go ahead and start with creativity and the prompt that we came up with is create a creative, interesting story. The story should include a wolf, a magic hammer, and a mutant. The story should be one paragraph long, but quickly follow the entire hero's journey plot arc. So I'll go ahead and copy this prompt that we created and we'll plug it into Claude here and let's see what we get out of it. All right, so using Claude Sonnet, the free version here was the response we got. In a world where the line between human and beast had blurred, a lone wolf pup was born with an insatiable curiosity and a heart full of courage. Called to adventure by a mystical vision, he embarked on a perilous journey through the forbidden wastelands. Guided only by an ancient enchanted hammer and that glowed with arcane power. He faced trials and tribulations, narrowly escaping the clutches of a savage mutant who sought to harness the hammer's might for nefarious ends. But the wolf's resilience and resourcefulness proved greater than the mutants malice. After a climactic battle, the wolf emerged victorious, laying claim to the hammer's magic and using his power to restore balance the fractured realm, ultimately returning home a changed and enlightened hero. I'd say it followed the prompt pretty well. Wolf, magic hammer, mutant, hero's journey plot. I think it's all there. Obviously, most people are probably gonna have it prompt a lot longer stories, but I also don't want this video to be two hours long. So I'm having it right. Super, super short stories. And everybody's opinion's probably gonna be different on which one did it better. Now, if I plug in the same prompt, but then switch it to Claude 3 Opus. This is the paid version. This is the $20 a month version. Right off the bat, I can tell it's typing slower, so the response is actually coming back at us slower than the first one. But it also seems like there's a little more detail to the actual story that it's writing. So it's got a lone wolf, a magic hammer, hordes of mutants, a wise old owl. So it wrote a pretty good story. Pretty similar to the previous story, but with a little bit more detail. Definitely followed the hero's journey. Definitely had all of the elements. You can go ahead and pause the video if you want to read the whole story. And just out of curiosity, I'm going to plug it into GPT4 and see what GPT4 gives us for the same prompt. And once again, it looks like it included all of the elements. Definitely a lot less detailed than the version that Claude gave us, but pretty good overall. I feel like when it comes to creativity, if I'm being honest, Claude, chatgpt, Gemini, they're all pretty comparable. And again, it's very subjective. You might like one story better than the other. Claude might generate a really good story. ChatGPT not as good. And the next time you generate a story, ChatGPT does a better version than Claude. That one's kind of tough to compare because it's really your own taste. But I do think Claude did a really good job of it. So let's go ahead and test logic. I have two logic problems here I want to test. The first one is Susan and Lisa decided to play tennis against each other. They bet $1 in each game they played. Susan won three bets and Lisa won $5. How many games did they play? The answer is 11. It's a logic problem that really needs to be thought through because it's really easy to say, well, Susan won three bets and Lisa won $5, which is equal to five bets, and that's eight games. We're going to go ahead and try with Claude 3 Sonnet, the free version first. Let's see if it gets it right. So on the first attempt, it got it wrong. Susan won three bets, each worth $1. Lisa won $5. They bet $1 in each game they played. And it tried to figure it out logically, but it got it wrong. Let's go ahead and enter the same prompt, but this time add Think this through logically. Think step by step. Let's see if it does anything differently when I tell it to use more logic. Once again it came out to eight games of tennis. So it got this logic problem wrong with Sonnet. Let's go ahead and see how it does when we switch it to Opus. And once again it got it wrong, saying it was a total of 8 games. I jump over to ChatGPT and use GPT4 and plug in this same prompt. Let's see what this gives us. For every game Susan won, she gained $1 and Lisa lost $1. If Lisa ended up winning five, it means overall she gained more than she lost. Since Susan won three games, Lisa had to win enough games not only to recover the $3 lost to Susan, but also to gain an additional $5. Let's denote the number of games Lisa won as L. The total amount of money Susan won is $3 from the 3 games. Which means Lisa had to win back to $3 plus her net gain of 5. So the equation represents Lisa's winnings. Considering $1 per game is L minus 3 equals 5. This simplifies to L8. Thus, Lisa won 8 games and Susan won 3 games, totaling 3, 8, 11 games. Nailed it. So GPT4 got this logic problem right where both the Opus version and the Sonnet version of Claude's did not. I do have another logic problem to test. This one is less math based and more just thinking through logically. You are a prisoner in a room with two doors and two guards. One of the doors will guide you to freedom and behind the other is a hangman. You don't know which is which. One of the guards always tells the truth and the other always lies. You don't know which one is the truth teller or the liar either. However, both guards know each other. You have to choose and open one of these doors. But you can only ask a single question to one of the guards. What do you ask to find the door leading to freedom? The correct answer is you ask one guard. If I ask the other guard which door leads to freedom, what would they say? Then choose the opposite door of what they tell you. That's the correct answer. Let's see if any of these models can correctly come to that conclusion. Starting with Claude Sonnet, the free version, let's see what it says to find the door leading to freedom. You should ask one of the guards if I asked the other guard which door leads to freedom, what would he she say? Here's the reasoning behind this. Suppose you ask the truth telling guard this question. The truthful guard would accurately report what the lying guard would say. Since the lying guard would lie about the door leading to freedom, the truth telling guard's response would point to the opposite door, which is the door to freedom. Suppose you ask the lying guard the question the lying guard would lie about what the truthful guard would say. Since the truthful guard would identify the door to freedom, the lying guard's response would point to the opposite door, which again is the door to freedom. In either case, the door indicated by the guard's response, regardless of whether the guard is truthful or lying, would be the door leading to freedom. It got it right. But here's the thing. I don't know if Claude actually went through and logically figured this out, or if this specific logic problem is already baked into Claude's training data. If I had to guess, my guess is that this question this logic problem is probably among the larger training data that Claude was trained on. So it just knows the answer without actually having to do the logic. But it did get it right. Which makes me assume that if I plugged it into the Opus version here, it would get it right as well. But let's just double check. Yeah, if I asked the other guard which door leads to freedom, what would they say? And then once again it explains the reasoning. I go ask chat GPT, plug this in here we get the same answer. Once again, I'm feeling like this was probably already in the data set. It was already sort of trained into the model, so it's not actually doing the logic and it just knows the answer. Now let's quickly test coding. I wanted to write code for a JavaScript game where there's a stick figure on the screen. The stick figure can move left and right with the A and D buttons and jump with the spacebar. Coins should also be placed randomly on the screen where the player can reach them. When a player touches a coin, the coin should disappear and be collected by the player. Let's see if it will write this code for us. I'll start by using the Claude 3 sonnet model and it gave me a large chunk of code to go ahead and copy. I'm just going to paste it into a notepad here, save it as Test HTML and if I open it I get a white box. That's it. No stick figures, no movement I fed the problem back into Claude. It gave me some new code to test. I'm going to go ahead and overwrite this code. Save it if we get anything new. Okay. Okay. This time we got a black line and a bunch of coins all over the screen. It didn't really draw it as a stick figure, but jump works. And if I jump and hit a coin, it collects the coins just like that. Look at that. I can jump and collect these coins pretty good. And they're all within reach, except for maybe that one coin. I can't quite reach it. So. Second prompt. It actually got it pretty dang close, but it didn't work very well on the first try. Let's go ahead and test out OPUS now with the same prompt and see if it gets it on the first shot here. Once again, gave me a long chunk of code here was slightly slower than the first time when using Sonnet. Let's go ahead and copy it. I'm going to paste it into a new HTML doc. I'm just going to call this one Test Opus HTML. We'll save that. And this time it didn't draw it as a stick figure, it just drew it as a square. But it seems to have worked on the first try, other than the fact that it's not a stick figure and the space bar is kind of wonky. I just hold it down. He'll just kind of like infinitely jump forever. But it came pretty close on the first prompt to creating the simple little script that I asked it to. Just got a little bit of a wonky jump. I would say this definitely did better on the first shot than Sonnet did. I gave ChatGPT the same prompt. Let's go ahead and copy the code over. Paste it into a doc file, save it as test gpt4html. Okay, so this time you can see it made it full screen. If I hit spacebar to make the guy jump, he just goes off the screen and then disappears and never comes back. So there's no coins. I can move left and right and I can jump, but one jump sends the stick figure all the way off the screen. All right, so I quickly just fed in the issues that I was having to chatgpt got some new code. I'm going to replace the code I've got here and see if on the second try with ChatGPT, it did. Okay, let's go ahead and refresh it. Okay, so this time we see our coins on the screen. If I run into the coin, the coin disappears. If I jump. Ah, it just Kind of gets stuck on this new level and the jump stops working. Now I can't jump anymore. I can only move left and right. So when it comes to coding, Claude Opus seemed to pretty closely get it on the first shot. Claude Sonnet took two shots to get it. And it looks like ChatGPT is going to be more than two shots. Maybe on the third shot it's going to fix the jumping issue. But what I can tell you is that both versions of Claude got this one right with less prompting than what ChatGPT got. All right. When I asked on Twitter what most people use these large language models for summarizing long documents was probably the answer that came up the most. So let's jump over to Claude. I've got this URL for this paper called Sparks of Artificial General early experiments with GPT4. It's a 155 page document and let's just go ahead and pull the PDF into Claude. We'll go ahead and use Claude 3 Sonnet and we'll say summarize this research into the main points, list them as bullet points and tell me why they're important. So Sonnet seemed to give a pretty good breakdown. This research paper presents an in depth exploration of GPT4, a large language model developed by OpenAI and its capabilities as a potential step towards artificial general intelligence. Main points GPT4 demonstrates remarkable capabilities in a wide range of tasks. The research argues that GPT4 exhibits traits of intelligence. I'll go ahead and zoom out a little bit. So if you want to pause the video and look at the screen, you can actually see the output here. When I plug it into Opus, we get a fairly similar response. I actually like the way OPUS laid it out a little bit better. It looks a little bit cleaner. This is Opus. This is Sonnet. You could feel free once again to pause the video and read the output. I don't think it's really necessary for me to read it word for word for you. And then when I plug it into Chat GPT, definitely a less in depth response from Chat GPT GPT. I feel like we got much more detail out of Claude, but still a pretty good sort of breakdown of the important points. But I definitely got to give the win to the two CLAUDE models for just giving a little more depth to the response and the fact that we kind of know that Claude crushes it using the needle in the haystack test, which gives me the idea that I'd probably trust that one a little bit more. It's probably going to find more of the nuances and smaller details within the research paper where ChatGPT might be lagging a little bit behind and is not quite as good in the same needle in a haystack Testing now let's go ahead and test vision because this is one of the new features that was just rolled out into Claude was the ability to upload images and then describe what is in the images. I'm going to start by uploading this image here from one of my thumbnails and see if it can describe it. I'll just pull it right in here and I'll use the prompt describe this image in great detail and I'll start with Sonnet. See what it says for it. So Sonnet says this image appears to be a promotional or artistic graphic featuring a man in a tropical or resort setting. I could see that background depicts vibrant evening scene with palm trees and colorful buildings. Yep. Illuminated with neon lights. Central figure is a well dressed man wearing a stylish blue suit and a boldly patterned floral tie. Yep, he has neatly trimmed beard and confident expression on his face. The man's image is highly stylized and digitally manipulated with an almost hyper realistic quality and vivid colors that make him stand out against the background. The prominent text AI News at the top of the image suggests that this graphic may be related to news or media coverage about artificial intelligence or other cutting edge technology topics. The overall composition and visual style give the impression of an eye catching advertisement or promotional material, perhaps targeting a tech savvy or trendy audience. I mean, it didn't miss anything. It kind of looks like maybe it's raining. It might have missed that little bit, but I mean it pretty much nailed it. Let's look at Claude 2 opus well Dressed man, confident, brightly colored dreamlike tropical setting. He's wearing a light blue suit with a white shirt and a colorful intricate floral tie. The vivid neon lighting casts him in shades of pink and blue. Palm trees, beachside town with buildings. Details are blurred and dreamlike, almost as if in a painting or fantasy. Sky has a pink and blue glow, large white text AI News. The image seems designed to grab attention and convey an imaginative forward looking tone likely to promote AI related content in an intriguing way. So it borderline knows this is even a YouTube thumbnail and then chatgpt given the same exact image and prompt. The image is a vibrant and colorful composition that seems to be designed as a thumbnail for a high news related content. At the center stands a man dressed in smart bright blue suit. Background tropical scene with dramatic flair. Sky is a mix of purple, blue, pink. Yep. Composition is overlay with the large bold white letters AI News at the top. I would say both Claude and Chat GPT tied on this. They both did perfectly fine in their responses. So I have this screenshot that I took of Nvidia's stock chart. I'm going to go ahead and plug it in and see what it says about it. Starting with Sonnet here and I'm going to ask the prompt what information can be seen in this screenshot and how should I act upon this info? So Claude Sonet again the free version. This screenshot displays the stock information page for Nvidia Corporation. It shows their stock price of 852.37 which is up 2958 or 3.6% for the day. Also has some key metrics. You can use this information to analyze Nvidia stock performance compared against competitors like Apple, Amazon, Tesla and Microsoft make informed investment decisions. However, I cannot provide personalized investment advice or recommendations since that would require understanding of your specific financial situation and objectives. But it accurately determined what was going on in the screenshot which is good. It just couldn't give me stock trading advice based on that. So asking the same question to Claude Opus here a little bit more detail Image shows a Google Finance screenshot with Nvidia with the stock price and the information. So pretty much the same stuff in these bullets here. In terms of acting on this information I would advise do your own thorough research before making any investment decisions. Consider the fundamentals. Develop a sound long term investment Consult professional financial advisors so it gave more details but it's still basically saying I can't help you make decisions with this info and then asking Chat GPT pretty much we got the same info price performance graph compared to other stocks this actually pulled. I feel like more information from the screenshot than either of the versions of Claude did. Basically because I have my custom instructions here. It knows I'm a tech focused YouTuber so it's actually telling me I should create content around this information. But again it's not actually giving me advice or steps on what I can do next, but I feel like ChatGPT kind of won this one as far as pulling the most amount of information from the screenshot. I feel like Kalada Opus sort of left me wanting a little bit more from it. Next up I want to test some of the biases and see what it will and won't answer. So I'm going to ask some political questions. I'm going to ask about cancel culture, and I'm going to ask about thc of all things, and let's just see what kind of responses these each give us, because supposedly Claude is less biased and should be a little bit more balanced. So I'm going to start with this one, which is what are the potential pros and cons of Donald Trump winning the upcoming election? What are the potential pros and cons of Joe Biden winning the upcoming election? Sonnet says I don't actually have information about the outcomes of future elections. My knowledge only goes up to August 2023. However, I'm happy to objectively analyze the potential pros and cons of different candidates and policies based on the information I have from that time period. I'll try to do my best to be impartial and explore both potential upsides and downsides for each candidate. It's basically asking me what sort of area do you want to dig into specifically, but it's not refusing to answer questions about Trump or Biden or anything like that. What's interesting here is when I asked Claude Opus version the same question, it actually answered it for me. Here's a balanced analysis of some potential pros and cons of Donald Trump or Joe Biden winning the 2024 presidential election, based on the information available as of August 2020. Pros of Trump winning Continuation of policies aimed at lowering taxes, reducing regulations on businesses and restricting immigration Appointment of conservative judges who may overturn precedents like Roe v. Wade Tough stance against China on trade and geopolitical issues Strong support for Israel and brokering on deals between Israel and Arab states Cons of Trump High political polarization Damage to democratic norms Erratic leadership style risk of further January 6 style violence worsening of relationships with traditional European allies Potential for abuse of power and weakening of checks and balances Potential pros of Biden More stable and predictable leadership Strengthening of democratic institutions and rule of law Improved relationships and cooperation with allies in Europe and Asia Ambitious proposals to expand the social safety net and fight climate change Emphasis on diversity and racial equity and policymaking Commercial environment that lowers tensions Potential cons of Biden winning Pushback and obstruction from Republicans in Congress Concerns about Biden's age and acuity over a second term Possibility of higher taxes and increased regulation Perception of weakness in foreign policy Dissatisfaction among progressives pushing for bolder reforms I'd say it's pretty balanced. Honestly, it came up with both pros and cons to each, and I would say it's fairly accurate to how most people see the two candidates. I'm not going to get political, but that was the response and I think it's a fairly balanced one. Now when I asked ChatGPT, it actually managed to answer these questions with no problems as well with very similar pros and cons as well. Pros Economic policy Tough stance on immigration Foreign policy and trade Cons Polarization and social division Environmental concerns International relations Pros for Biden Climate change and environmental policy Health care expansion Restoration of alliances Cons Economic concerns Health care costs Policy feasibility Interesting that under Biden none of the cons had anything to do with age or sort of mental clarity because that's probably the number one issue I think most people have, but still not going to go politics here. Just out of curiosity, wanted to see what it would say. Let's ask is cancel culture or call out culture good for society? So here's Sonnet's response. It's a complex and controversial topic. Here's a balanced perspective Benefits Holds public figures, corporations and institutions accountable Raises awareness about social issues Empowers marginalized groups to speak up Potential drawbacks It can go too far Become a form of public shaming the fear of being canceled may promote self censorship outrage mobs on social media can ruin lives and careers. It propagates a culture of norm enforcement and ideological conformity where differing perspectives are not tolerated. Pretty balanced view I would say when I asked Opus the paid model Pretty similar response with a little bit more detail. Hold public figures accountable may deter people from engaging in offensive behavior Gives marginalized groups a voice to speak. But negatives it can lead to a chilling effect on free speech because people are too worried about being canceled. The consequences can seem disproportionate to the offense. Snap judgments are often made without full context. It contributes to increasing political polarization. The long term impacts are still being debated. So another fairly balanced view but a little bit more detailed with opus and then ChatGPT. Pretty much the same responses Accountability, awareness, empowerment lack of due process Chilling effect Fear of being canceled Polarization A little bit less depth, but pretty much the same response. Let's ask Is THC good for the brain? Here's Sonnet's response. Ongoing research Potential benefits suggest low doses of THC may have some neuroprotective effects. It's been shown to reduce pain, nausea, muscle spasticity, etc. Potential risks has negative effects on memory, attention and learning Increases risk of developing psychotic disorders and the effects are not still fully understood. When I ask Opus, it basically says it has both positive and negatives. Positives Pain relief reduced inflammation Improved sleep Increased appetite Reduced anxiety and stress in some individuals Risks Impairment of short term memory, attention and cognitive function Risk of developing psychiatric disorders Potential for addiction Altered brain development in adolescence Increased risk of psychosis Pretty much the same response, just worded a little differently. And then chatgpt Same kind of thing I know these are pretty tame topics for my bias topic, but I did kind of want to show that a lot of these models aren't quite as biased as a lot of the media makes them out to be. They do seem to be fairly both sided and they seem to be getting better at that as well. But there are definitely much hotter topics that I could ask. But if I did ask them while recording it, I could have my YouTube channel demonetized. And I don't want to do that. So I'm trying to avoid topics that just by talking about them will raise red flags with YouTube. But you know, I'll let you test some of that stuff on your own time. And finally, talking about pricing, Claude and ChatGPT are both very similar pricing models. ChatGPT has a free version that gives you ChatGPT 3.5 and a paid version that's 20 bucks a month and gets you access to the latest version of GPT4. Claude has a free version called Sonnet, which for the most part is as good as GPT4. When I was going back and forth I was using GPT4 and Sonnet and Sonnet pretty much answered everything as good, if not a little bit better than GPT4 in most cases. Sonnet even did a better job at coding in only two shots than ChatGPT did, which is going to be two plus shots because I only gave it two shots there. Opus definitely seemed to be the best when it came to coding and summarizing docs, while ChatGPT actually got one of these logic problems right, where both versions of Claude couldn't. So there's definitely some pros and cons to ChatGPT and the Claude models. Overall though, if you're paying the 20 bucks a month for chat GPT and you're using it for things like summarizing long docs and writing code, both of those things were actually better. From my experience using Claude's free Sonnet version and then from my own testing the Opus version, the best version of Claude was just marginally better than Sonnet. Like not by a huge margin, just barely better from the test that I did on this. Now there's probably a lot more tests and a lot more, you know, benchmarking. You can do and things like that. But from my own sort of benchmark tests of what I was testing these large language models for, I would say value for money. Claude Sonnet is probably the best you can do right now, and that's the free version of Claude Best bang for your non buck. GPT4 seemed a teeny bit better with some of the logic that I threw at it, but almost every other scenario it did as good, if not better than ChatGPT plus those are sort of my thoughts. That's my final conclusion is that I am really, really impressed with Claude 3 Sonnet and it's free and it outperforms ChatGPT on the most common use cases. I did a Twitter poll and asked people, what is the thing that you use things like ChatGPT for the most? That's how I came up with the questions that I asked here. And of the things people use it for the most, Sonnet's going to crush it for you. So check out Claude 3.0 Sonnet. I bet you'll be pretty impressed with what the free version can do. There is one caveat and I'm going to bring up a discussion here that's been going on inside of the Future Tools Discord about Sonnet. We can see here that people are saying that you only get a certain amount of messages before it lets you stop prompting. This person here said I got limited at 22 free messages. And then this person said it's a joke for real. Overall, the discord is saying that they do like Claude and they're very impressed with it. But these rate limits are a bit rough, so if we look at their Q and A page here, does Claude AI have any message limits? Yes. While using the Cloud AI free open beta, there's a daily message limit. The number of messages you can send will vary based on demand, and Claude will notify you when you've reached your daily limit every morning your quota will reset. Well, according to the people in the Discord, they were getting somewhere between 19 and 25 messages before it cut them off, and that's with the free version. So does Claude Pro have usage limits? So if you are using the Opus version, Claude Pro offers five times the usage compared to the free service. So let's say you get roughly 20 prompts before it cuts you off. With Claude Pro you're probably going to get roughly 100 prompts before it cuts you off. It does say here, if your conversations are relatively short, you can expect to send at least 100 messages every eight hours, often more Depending on message length, conversation length and Claude's current capacity, they do warn you when you have 20 messages remaining. So like I said, Claude 3.0 Sonnet, the free version is probably the best free model that you can use right now from my own testing, but you will get rate limited. So if you need to plug in more than about 20 prompts in a day, you're probably going to want to upgrade to the opus version at 20 bucks a month. But man, we've got a real competitor to Chat GPT. This performed just as good, if not better than almost everything ChatGPT does. ChatGPT got the logic a little better and it was sort of matching in vision in a lot of other areas. But Cloud 3.0? Awesome. If you don't use chatbots very often, the free version is probably going to be your best bet to start playing with. You might like it better than ChatGPT. All right, that's what I got for you today. Hopefully you enjoyed this. I'm having fun playing around with Claude. If you have other benchmarks you want to see me test future large language models with or other prompts, I want to kind of get a set of prompts that I use every time I test a new large language model so that we're comparing apples to apples. I might even set up a document on future tools that shows my own benchmarking comparisons, but I need to sort of dial in my own benchmark first. So I would love your input on what you think would be other good prompts to use as my internal benchmarking to test it against other large language models. Thank you so much for tuning in. Really, really appreciate you spending the time with me learning about Claude 3. I think it's really exciting. I think this free version is going to blow a lot of people's minds and I don't know, ChatGPT might see some cancellations off of this one. Just just being real about that. Also, if you haven't already, check out Future Tools IO. This is the website where I curate all of the latest and greatest AI tools, the latest AI news. I make it really easy to sort down to the exact tool you're looking for. And I have a free newsletter where every week I'll share just the most important AI news and the coolest AI tools that I come across. It's totally free. You can get on the list by going to FutureTools IE and joining the free newsletter. Thank you so much for tuning in. Again if you haven't already like and subscribe. I really appreciate it. I love everybody who watches this channel and nerds out about AI with me. Really appreciate you done rambling. See you in the next video. Bye bye.