CLAUDE 3 Just SHOCKED The ENTIRE INDUSTRY! (GPT-4 +Gemini BEATEN) AI AGENTS + FULL Breakdown

So something actually shocking did happen in AI today, and that was Anthropic's release of the next Generation of Claude Claude 3. Now this was a model that took everyone by surprise because it beats every other AI model across the board on the main benchmark. So without wasting any more time, let's get into the meat of this release. So you can see here that they actually released three new models. And in the Claude 3 model family there is Claude 3 Haiku, Claude 3 Sonnet and Claude 3 Opus. And essentially you can see as the models increase in intelligence, the cost does go up quite slightly. But these three different models are very, very fascinating. And later on in the video I'll show you how they all differ and how Claude 3's opus, the one right here, how this model is smarter than any other AI currently available, making it state of the art. You can see that they said a new standard for intelligence, which opus, our most intelligent model, outperforms its peers on most of the common evaluation benchmarks for AI systems, including undergraduate level expert knowledge, graduate level expert reasoning, basic mathematics and more. It exhibits near human levels of comprehension and fluency on complex tasks, leading the frontier of General Intelligence. All Claude 3 models show increased capabilities in analysis and forecasting, nuanced content creation, code generation and conversing in non English languages like Spanish, Japanese and French. And now this is where we actually get to the crazy benchmarks, because you're about to see something that surprised me and caught me off guard. These benchmarks are actually quite shocking. You can see that Claude 3, their most powerful model, Opus, actually surpasses the other state of the art models. You can see that GPT4 and Gemini's recently released 1.0 Ultra. And actually I wouldn't say pale in comparison to the new model, but they actually do get surpassed on these benchmarks. Clearly you can see here that the undergraduate level knowledge, the MMLU, is at 86.8% compared to the other models. It is beaten. And what's crazy is that we can see that on the left across the board, this is excelling at every single tasks. And this goes to show how crazy things are, because it was only recently that we did just get Gemini Ultra, which surpassed GPT4 on every single benchmarks. But now we literally just get Claude 3 only around two to three months later that surpasses Gemini 1.0 ultra on every single benchmarks. You can see that this is some really, really impressive stuff because if we take a look at the percentages, we are nearing 100% in some categories. You can see 95.4% on common knowledge, LS swag, and we can see on some of these other ones that it's 96.4%, 90.8% and 95%. So this is something that really did shock me because I didn't expect a CLAUDE release for a little bit more. But not only did they surprise me about the release date, these benchmarks surprisingly managed to take on Google already, which is a very impressive feat. And dethrone GPT4. Now, something that I did see, and this is of course around the first hour of this model being released, was that there is a qualitative aspect that you can't sort of look at when you're looking at these benchmarks. Essentially, what I mean here is that whilst it is good to, yes, look at these benchmarks and think, okay, how does the AI do on these, on these maths problems, on these coding problems and reasoning over text, but the actual qualitative data that you can get from your users is something that is very important because at the end of the day, it's your users who's going to be using this that determine whether or not your product is actually good. And so far, based on what I've seen, the, the qualitative data where people are actually talking about how good the model actually is, shows us that this model isn't just good at reasoning and doing well on some of the benchmarks. This is clearly a model that people really do like. You can see here that this person says, excited to share what we've finally been working on. To me, talking to Opus feels different than talking to any other large language model. It seems just to get you, this can't be represented in any evaluation metric or benchmark. You have to just go experience it yourself. And this person says, evaluations aside, clause three, OPUS feels like the smartest model I've talked. So that is something that I feel will be important. And I would love to see where this model ends up on the LLMs chatbot arena, because that is something that uses qualitative data to judge where the models actually do lie. And I think it is a very important metric. One of the most surprising things that we did get from the Claude 3 release was, of course, the vision capabilities. Claude 3's different models, and I will get into later how these models actually do differ, do possess new vision capabilities. This is actually a model that is multimodal. It states that the Claude 3 models have sophisticated vision capabilities on par with other leading models. They can process a wide range of Visual formats, including photos, charts, graphs and technical diagrams. We're particularly excited to provide this new modality to our enterprise customers, some of whom have up to 50% of their knowledge bases encoded in various formats, such as PDFs, flowcharts or presentation slides. So you can see that Claude 3 is finally becoming a model that is very, very effective at a wide range of tasks, not just text. And something that is important is they've actually shown us a demonstration on how well Claude 3 opus, combined with the Vision model is going to be at doing.

In this video, we're going to see if Claude and a couple of friends can help us analyze the world economy in a matter of minutes. Okay, I've asked Claude 3 Opus, which is the largest model in Anthropic's new Claude 3 family, to look at the GDP trends for the US and write down a markdown table of what it sees. We've given OPUS and all the other models in the Claude 3 family extensive training on tool use. And one of the major tools it's using is this webview tool. It goes to a URL, looks at what's on the page, and because it's multimodal, it can use the information on that page to to solve complex problems. So here's the markdown and it's important to note that CLAUDE doesn't have direct access to these numbers. It's literally looking at this same browser you and I are seeing, looking at the trend line and trying to estimate what the exact numbers are. Let's see how accurate it was. We've asked the model to create a plot of the data and it's used the second tool, this Python interpreter, to write out the code and then render the image for us to check. And here's the image, look. It's actually added helpful little tooltip animations to explain some of the major peaks and troughs in the last decade or two of the US economy. And we can compare that graph with the actual data. And it turns out it's pretty close. It's actually within 5% accuracy. And by the way, Claude's transcription here isn't just coming from its pre existing knowledge of US gdp. We tried it with a large sample of made up GDP graphs and its transcription accuracy was within 11%. Next we asked the model to do some statistical analysis, projecting out into the future, performing simulations to see where the GDP of the US might head. And we can see that it's run this analysis using Python and it's able to perform these Monte Carlo simulations to see what the range of GDP possibilities might look like for the next decade or so. But I wonder if we can go further. We're going to get the model to analyze a more complicated question, that is how GDP might change across all of the biggest world economies. And then to help it do that, we're going to give it one more tool called Dispatch subagents. This basically allows the model to break down the problem into lots of sub problems and then write prompts for other versions of itself to help pick up the slack. The models can then complete a more complex task by all working together. Here you can see it's written this prompt and given very precise instructions that it wants the other models to follow, including a format for the data that it's hoping to return. It's dispatched a version of this prompt to one model that's going to look at the us, one for China, one for Germany, Japan and so on. We can see in these progress bars that the subagent models are now completing the set task for each of the individual economies. They're going to the relevant web pages, they're getting the information, they're running the code to analyze it, just like we saw in the previous US example, but all in parallel. Let's just skip forward to see what the model produced. You can see it's run the analysis. It's produced a pre and post pie chart of how it expects the world economy to look in 2030 versus 2020. And it's given us a written analysis too where it makes verbal predictions that relate to the statistical analysis that it ran. It's telling us that it thinks the GDP share of particular economies will change and which ones will be larger or smaller by 2030. So there we have it. Complex multi step multimodal analysis run by a model that can create subagents to get even more tasks running in parallel. We're excited to see what you, our customers can do with these advanced Claude 3 capabilities.

So yeah, that small demo was rather impressive. We actually got to see Claude 3 act as not only someone who is able to accurately take data from an image that doesn't exactly have the data, just doing pure estimates, which is really, really good and just shows how well their visual system is. But also a very, very interesting feature that caught me off guard were two of them. The first one being this simulations. I think that this is really, really cool. We can see it doing some kind of, you know, kind of tree search which looks absolutely amazing and is going to be very, very useful for data analysis. Like they stated, it's going to be used for predicting things. And I do wonder how some of those predictions will hold up in the future. And you know, if we actually do use the future data and then say, look, this actually was correct, maybe we could even get models that are increasingly, increasingly smarter as their predictions come true, they decide to, I guess you could say, reinforce that data somehow. So that is going to be a whole new area that I really didn't see before, that I'm excited to explore with Claude 3. Now I know most people are going to be excited about this, and so was I. The sub agents area was something that I found to be absolutely astounding. The this is where you can literally get an AI model to automatically decide to dispatch sub agents to do the rest of the task. And I just find that that is a concept that is really, really effective. And you can see here that it managed to complete the task with much more efficiency than just asking one model. So I think that what we have here on that demo was showing us just how great this Claw 3 model is, not just in terms of its common sense reasoning, in terms of its vision capability abilities, but also in terms of its ability to do complex step by step reasoning with multiple different tasks. And that right there within the API and tool use which they state is coming soon, I'm guaranteeing you, is going to have some massive implications for the industry because people are going to be using this in very, very creative ways, as we've seen with other AI models. Next, of course, we do have another short demo and this is by their other model, Haiku, and this one is very, very fascinating too.

Claude Haiku is one of the fastest and most affordable vision capable models in the world. To demonstrate this, we're going to read through thousands of scanned documents in a matter of minutes. The Library of Congress Federal Writers Project is a collection of thousands of scanned transcripts from interviews during the Great Depression. This is a gold mine of incredible narratives and real life heroes, but it's locked away in hard to access scans of transcripts. Imagine you're a documentary filmmaker or journalist. How can you dig through these thousands of messy documents to find the best source material for your research without reading them all yourself? Since these documents are scanned images, we can't feed them into a text only LLM and these scans are messy enough that they would be a challenge for most dedicated OCR software. But luckily Haiku is natively vision capable and can use surrounding text to transcribe these images and really understand what's going on. We can also go beyond simple transcription for each interview and ask Haiku to generate structured JSON output with metadata like title, date, keywords. But also use some creativity and judgment to assess how compelling a documentary, the story and characters would be. We can process each document in parallel for performance and with Claude's high availability API, do that at massive scale for hundreds or thousands of documents. Let's take a look at some of that structured output. Haiku is able to not just transcribe, but pull out creative things like keywords. We've transformed this collection of many, many scans into rich keyword structured data. Imagine what any organization with a knowledge base of scanned documents, like a traditional publisher, healthcare provider or law firm can do. Haiku can parse their extensive archives and bodies of work. We'd love for you to try it out and see what you build.

So once again that was a very impressive demo on how these vision capabilities can be used at scale for multiple industries in multiple applications. And once again I can't imagine what people are going to do with once they do get their access on the APIs. Now another thing that was actually really cool was that they stated that there is going to be immediate response from one of the CLAW three models and that is of course their most lightweight model, Haiku. So it states that Haiku is the fastest and most cost effective model on the market for its intelligence category and it can read an information and data dense research paper on ArcSive, which is around 10,000 tokens with charts and graphs in less than three seconds. So following launch we expect to improve performance even further. And for the vast majority of workloads, sonnet is two times faster than Claude 2 and 2.1. These are the previous models with higher levels of intelligence and it excels at tasks demanding rapid responses like knowledge retrieval or sales automation. And Opus delivers similar speeds to Claude 2 and 2.1, but with much higher levels of intelligence. So you can see right there, these near instant results are going to be able to provide some very, very interesting applications because as you all know, AI that is very latency laden, I guess you could say isn't something that is quite effective in certain responses, for example in live chats, auto completions and in certain scenarios where responses must be immediate and real time. Haiku being the most cost effective model and the fastest is going to be interesting to see if it actually does manage to dethrone some of the other ones who are very, very quick. But considering the intelligence, it might just be Haiku that takes the cake. There's another very impressive demo that I Do want you to see. And this is their model Sonnet acting as a language part.

This is just a simple prompt to turn Sonnet into a dialogue agent that will talk with you in a language that you're trying to learn. So I chose Spanish, and I wanted it to basically take my imperfect Spanish and help me improve it. So I decided I wanted it to do a few things. I wanted it to take my message, which will be in kind of imperfect Spanish, and. And write out what it thinks I intended in English. I then ask it to write back the ideal learner message, which is just my message, as it kind of should have been written in Spanish. So I can see the kind of ideal form of this. Then I ask it to write a teacher response which just responds to me in Spanish so that I can continue the conversation. Great. So this is basically just Sonnet saying that we're ready to start. So I'll just start with a simple first message. And here it's following the format that I asked for. So it's repeating the message that I tried to send it back to me in English. It's telling me how I should have said it. So it's corrected some of the grammar issues in my request, and then it's responded to me in Spanish, and then it's asked me where I'm from. So, okay, now imagine I don't know a certain word in Spanish, but I still want to say it. I'm going to just include that word in English in square brackets, and hopefully it'll just translate it back to me. And suppose I hit a roadblock because I just don't understand the message that it sent to me. I can just ask it to translate that message to me into English, and then I can read that and I can respond to it again in Spanish, continuing the dialogue. And as a final step, you could ask Sonnet to create a little quiz for you based on the things that you've been discussing. So hopefully that is a useful prompt if you're interested in using Sonnet as a language learning partner. I hope you try it out.

Now, something in addition that I did also want to mention was less refusals. You can see here that Claude 3 actually refuses things a lot less. It states that previous CLAUDE models often made unnecessary refusals that suggested a lack of contextual understanding. We've made meaningful progress in this area. Opus, Sonnet and Haikyuu are significantly less likely to refuse answer prompts that border on the system's guardrails than previous generations of models. As shown below, the Claude 3 models show a more nuanced understanding of requests, recognize real harm, and refuse to answer harmless prompts much less often. This is definitely something that I would say is a win for CLAUDE and Anthropic because one of of the main problems with Claude 2.1 was that it really just didn't answer many of your questions. Probably most of the times it just refused to answer your questions, leading to user frustration. And I was someone who was actually in that group because trying to use CLAUDE when it's very, very good is very, very fun. But when it doesn't want to respond to what you're stating because it thinks there's a real danger when there really isn't is one of the most frustrating things you could ever experience. But you can see they've now improved this Another thing was of course, the improved accuracy. They state that businesses of all size rely on our models to serve their customers, making it imperative for our model outputs to maintain high accuracy at scale. To assess this, we use a large set of complex factual questions that target known weaknesses in current models, and we categorize the responses into correct answers, incorrect or hallucinations, and admissions of uncertainty where the model says it doesn't know the answer instead of providing incorrect information. Paired to Claw 2.1 opus demonstrates a twofold improvement in accuracy or correct answers to on these challenging open ended questions, while also exhibiting reduced levels of incorrect answers. In addition to producing more trustworthy responses, we will soon enable citations in our Claude 3 models so they can point to precise sentences and reference material to verify their answers. Now there was also something that I really wanted to talk about, which is the perfect recall. Claude 3 Opus has recall accuracy that borders on 99%. They state that the Claude 3 family of models will initially all offer a 200k context window upon launch, and however, all three models are capable of accepting inputs exceeding 1 million tokens, and we may make this available to select customers who need enhanced processing power to process long context prompts effectively. Models require robust recall capabilities. The Needle in a haystack evaluation measures a model's ability to accurately recall information from a vast corpus of data, and they state that we enhance the robustness of this benchmark by using one of the 30 random needle question pairs per prompt and testing on a diverse crowdsource corpus of these documents. Claude 3 Opus not only achieved near perfect recall, surpassing 99% accuracy, but in some cases it even identified the limitations of the evaluation itself by recognizing that the needle sentence appeared to be artificially inserted into the original text by A human. So you can see here that Claude 3 is actually a very, very effective system that is able to completely identify what's going wrong. And it's a 200,000 context window. And they also state that all of these models capable of 1 million token context input. Which just goes to show the era of one million context window is upon us. And this is very, very impressive stuff because this now allows for a lot more use case capabilities. Now one of the main questions that I actually had when looking at CLAUDE was of course the difference between the three models. This was something that was a little bit confusing when looking at a blog post initially, but they've actually made it really simple to understand. If you want the Cliff Notes, you can just screenshot this and share it to whoever might need it. But essentially OPUS is just the highest intelligence available. It's the smartest model that you use if you're trying to get the most factual answer. Sonnet is strong performance at a lower cost. So it's a balance of intelligence and of course the cost. And Haiku is the near instant speed and a very low cost. So that is essentially the difference between these three models. If you were wondering now we can dive into a little bit more detail on these models. You can see that Claude 3 opus and then you can see at the bottom here it does have the differentiator. And the differentiator is the main thing that you do want to pay attention to. This is just higher intelligence than any other model available. You can see it costs. The costs do seem quite steep. If I'm being honest with you guys, that does seem like a pretty expensive model. But then again, this is a state of the art AI system that is leading the frontier of AI intelligence. So there's no surprise that this model is that expensive. Now of course it shows us the potential use is for opus and it does say task automation, plan and execute complex actions across API and databases. Interactive coding, research and design, research review, brainstorming and hypothesis generation, drug discovery. Of course we've additionally got strategy, advanced analysis of charts and graphs, financials and market trends, forecasting, and then of course the highest intelligence of any other model. And this is opus. Now then of course we do have Sonnet, which is where they state Claude 3. Sonnet strikes the ideal balance between intelligence and speed, particularly for enterprise workloads. It delivers strong performance at a lower cost compared to its peers. And it's engineered for high endurance in large scale AI deployments. It says that the potential use cases for this are data processing. It says Rag or search and retrieval over vast amounts of knowledge. Sales, product recommendations, forecasting, targeted marketing and time saving tasks such as code generation, quality control and pass text from images. Of course, like I stated before, it's more affordable than other models with similar intelligence and it's better for scale. So if there is a model with similar intelligence as Sonic, this model is just a bit cheaper. And of course we do have haiku and Claude 3. Haiku is our fastest, most compact model for near instant responsiveness. It answers simple queries and requests with unmatched speed. Users will be able to build seamless AI experiences that mimic human interactions. So of course the potential use cases for this are customer interactions, quick and accurate support in live interactions and translations, content moderation, catch risky behavior or customer requests, and cost saving tasks such as optimized logistics, inventory management, extract knowledge from unstructured data, and essentially smarter, faster and more affordable than any other models in its intelligence category. So overall we can see that this new state of the art system, Claude 3 by anthropic has really surprised everyone and taken us back because it is something that is now a state of the art model that surpasses every other AI system. So this is something that is quite fascinating because the AI space is always rapidly evolving and it was only a couple of months ago that we had a new AI system literally surpass GPT4 and now that system has been surpassed. So it seems as if the race is heating up and things are accelerating. But I leave you all with this question. What do you think about Claude's new model? Are you going to be testing this out? Because of course right now you can see that you can actually use Claude if you want to, if you want to be able to just test out how good the system is. And I'm going to be doing another video on actually testing out the system, showing you some of the best use cases, because I feel like that is better saved for another video rather than talking about this actual announcement. So hats off to the Anthropic team for an amazing product and hopefully we can all have some fun using this amazing new AI system.