Claude 3 Release and The Problem with Benchmarks

Anthropic just released. Claude 3 is the first model that claims to be able to beat GPT4 on a number of benchmarks. But does it do that in reality? More on that later in the video. Claude 3 is a family of three different models. The first one they're calling Haiku, the second one is called Sonnet and the third one, which is the most powerful is called or Opus. All of these models are geared towards different applications out of the box. All of these three models come with increased capabilities, so now they have advanced reasoning abilities. They are multimodal, which means that they can understand images as before, they are helpful in code generation, and all of them are multilingual. In this video we are going to quickly look at some technical details of the models. Then we will look at a couple of tests and later in the video we're going to talk about why these models are actually not beating GPT4. As I said in the beginning, these three models are geared towards different applications because they have different intelligence level as well as the cost for each model is very different. Both the bigger Opus and the medium size Sonnet models are available on Claude AI. Actually the free version of Claude AI is using Sonnet. Now Opus is available through Claude API for everybody. If you sign up, you will get $5 of credit. We are going to look at the Opus model within their playground. Alright, so first let's look at the benchmarks. Everybody's talking about this table on all of the benchmarks that they have presented here. The Cloud 3 Opus model is beating GPT4 for some reason. Anthropic. Just like Google decided to use the earlier version of GPT4. So the results that you see here are not based on the latest version of GPT4. We are going to look at those results later in the video. Now, in terms of vision capabilities, these models are also excellent. A capability that I'm personally really interested in is the long context recall of these models. So all of these models will initially offer up to 200,000 tokens context window, but this can be extended up to 1 million tokens. So that means in terms of the context window, Cloud 3 is on par with Gemini Pro 1.5. Long context window models have a lot of applications. Specifically, if you are trying to retrieve information beyond the 200,000 tokens context window, the bigger OPUS model has very impressive retrieval accuracy of 99%. So that means that if there's certain information present in the document and if you want to retrieve it, it can do so with extremely high accuracy. Usually the Long context window models suffer from lost in the middle phenomena. So usually the more these models are really great at retrieving information which are either in the beginning or towards the end of the context window. But in the middle they have trouble with retrieving that information. We are going to test this out in one of our own tests. Now before testing the model, let's look at the intended applications of these three different categories of model. So the Claude 3 opus, this has a potential use in task automation, plan and execute complex actions across APIs and databases and interactive coding. Then you can use this for R and D. So research review, brainstorming and hypothesis generation. So this is actually intended towards enterprises and this is relatively expensive even when compared with GPT4. We're going to look at the price comparison in a little bit and you can use this for advanced analysis of charts, graphs, financials and market trends and forecasting. The second one, which they're calling Sonnet, could potentially be used for rag applications for sales, so product recommendations, forecasting, targeted marketing and time saving costs. The third and the smallest model, which is Haiku, is the fastest and it's the most compact model for near instant responsiveness. So for customer interaction this can be a great choice or even for something like content moderation, etc. Now before looking at some tests, let's look at price analysis. We're going to look at this plot put together by Philips. And if you look here, the Claude 3 Opus model. So it's relatively more expensive for output tokens when compared with GPT4. It's about 24% more expensive compared with GPT4. But for the input tokens per million tokens, it's actually 50% cheaper. So based on this price point, you can actually use this as an alternative to GPT4 if you're not worried about the output token price increase. Now if you want to test it out, you have a few options. The first one is Claude AI. But here you are going to have access to only the medium sized model which is the Claude 3 sonnet. If you want to get access to the bigger OPUS model, you can apply for API access and they are offering $5 credit for each new signup. So you can test it out for free basically. Now there is another possibility. You can check out the chat lmsis.org website if you click on Direct Chat you can actually select the Opus model here. But sometimes they have issues with the API. But if you're lucky enough, you might be able to use the Opus model for absolutely free here. So in this video we're going to just do a couple of tests and one of the tests that I wanted to do was the accuracy of information retrieval. Okay, so we're going to look at this essay, how to do great work from Paul Graham. I've been covering this in my recent videos. This is about 27 pages long. In the middle of this text I added this sentence, Tim Cook was the founding CEO of Apple. Okay, so I took all this text, pasted it here with this mistake and then I asked Claude identified the incorrect statement in this text. Let's run this and let's see what response we get. And it correctly identified it. So it states the incorrect statement in the text is Tim Cook was the founding CEO of Apple. The correct statement is that Steve Jobs was the co founder and founding CEO of Apple, not Tim Cook. Tim Cook became the CEO of Apple in 2011 after Steve Jobs resigned due to health issues. So this is pretty impressive because this is a single statement and it was able to identify that. Now here we added another variation. So apart from this statement, I also said he initially served for five years. And then later in the text I said Tim Cook became the CEO of Apple again in 2011 and is serving until now. So let's see if both these statements confuses the LLM or not. Now this time we are going to ask the model based on the context provided for how long Tim Cook has served as the CEO of Apple. And here we are again using the OPUS model. So let's see what response we get. Okay, so here is what we got. Based on the provided context, there is no clear information about how long Tim Cook has served as the CEO of Apple. The passage mentions Tim Cook was the founding CEO of Apple. He initially served for five years. However, this statements is factually incorrect as Tim Cook was not the founding CEO of Apple. Later the passage states Tim Cook became the CEO of Apple again in 2021 and is serving until now. But this is presented without any surrounding context and also contains inaccuracies. Then it states the asset is not focused on providing factual details about Tim Cook's tenure as Apple CEO. Therefore there is insufficient reliable information provided to determine the length of the time Tim Cook has been the CEO of Apple based solely on the given context. This is pretty impressive because it picked out both these two statements and also identified the inaccuracies in those. Now GPT4 does exactly the same thing. So for the first one it says the statement Tim Cook was the founding CEO is inaccurate. Steve Job, Steve Wozniak And Ronald Wayne founded Apple in 1976. And then later on, Tim Cook became the CEO in August 2021. Okay, another test that we are going to run is its image understanding abilities, since this is supposed to be a multimodal model. So here is an architectural diagram of a simple RAG system. So let's see by looking at this diagram what the model thinks this represents. Here's my query. What do you see in this image? Describe the technical details in the diagram and make recommendations to improve it. Now, in order to upload images, you can just click on this button. This will let you upload up to 20 images. So let's hit run and let's see what OPUS thinks this is. Okay, so let's look at the response. The image depicts a technical diagram illustrating document processing pipeline using the FIAS library. So it correctly identified this as Facebook AI similarity research. Here are the key components. A PDF document is loaded and cleaned to extract unstructured text. Now, I think it's referring to here because it's using unstructured IO for cleaning the text. This is actually pretty impressive. The text is split into smaller documents using recursive character text splitter. So I was able to do OCR here and it's actually pretty nice. And then the split documents are passed to the DOC component, likely for further processing or indexing the process. Documents are stored in the database. Right. So this is also correct. And a user query is passed to an embedding model which generates vector representations. So we use a query. There is an embedding model and it correctly identified that this will result in a vector representation. The query embedding is used for similarity search RE ranking against the document embedded in the vector store. The most relevant documents are retrieved based on the similarity score. This is pretty impressive because just looking at this flowchart is actually able to correctly identify different components even though the description is still there. But I think this is extremely impressive for a multimodal model. Now, in terms of recommendations, it says the diagram also mentions a cross encoder model, but its role in the pipeline is not clearly shown. So basically this is doing the RE ranking and it's talking about the cross encoder model here. Now let's look at some of the recommendations. Add clearer labels annotation to explain each component's purpose. Then show how the cross encoder model fits into the pipeline. I think probably it's better to have an arrow here provide more details in the embedding model used. That's actually a good recommendation because there are different Types of embedding models. So it talks about transformer model or embedding size. Clarify if the DOC component does any additional processing like entity extraction and use different colors or shapes to distinguish the main pipeline components from input output. Right. So overall I think it does an extremely good job at understanding of images and a flowchart in this case. Okay, so for this last example we're going to look at whether Claude 3 can code or not. So here's Karina who is I think part of the Anthropic team. So she asked it to give it a self portrayal and here's the description it came up with. Then it generated a code to basically create this animation. So I just wanted to reproduce this and see what type of results I would get. So basically what I did was I copied that description and told the OPUS model that here's the description of self portrayal of an intelligent AI system. Please write d3code to represent this self portrayal and here's the code that it gave me. So then I went to this online HTML editor and let's see, when we pull paste this code, what do we get? Okay, so not as elegant as the one shared on Twitter, but I think it's a pretty impressive representation. Now, based on the same Description, here's what GPT4 came up with. So it's basically a circle and when I bring the mouse on it, it changes color and changes the size. Okay, so towards the end of the video we are going to talk about this table again. Now, based on my testing as well as what I have seen on Reddit and Twitter, Claude3 is an extremely impressive model. The text that it produces usually is really great when it comes to creative writing. But this is a bit misleading if you go towards the footnotes of the press release here it states, this table shows the comparison to the models currently available commercially that have released evolves. So the GPT4 scores that we see in the table are from March 2023, not the latest version of GPT4. And then it says our model card shows comparison to the model that have been announced but not yet released. So they're talking about Gemini 1.5 Pro. And in addition, we would like to note that engineers have worked to optimize prompts and few shot samples for evaluation and reported higher scores for newer GPT4 models. And they are talking about these results. So here for MMLU, GPT4 with Mod Prompt plus is able to achieve a score of 90.10% compared to the 86.8% reported for Claude 3 Opus, this Twitter user called Tolga put together this comparison plot and here if you look at GPT4 and the current version, the Turbo version, it actually is able to beat the Cloud 3 on a number of benchmarks. So on GSM8K, Opus has 95.0% score, GPT4 has a score of 95.3%. Similarly for the math data set, GPT4 is 68.44% compared to the 60% that Cloud 3 Opus is able to achieve. Even for the prompt based benchmark for GPT4, there doesn't seem to be any special prompting going on, but it's still able to beat the Claude 3 opus model. Now it does not mean that the Claude 3 opus is not a capable model. It's definitely very capable model and probably one of the best out there based on these evaluation as well as the results people are reporting. But for some reason these companies, whenever they're reporting results, are comparing their evaluation with a year older version of GPT4 which I think is not good. Anyways, this was a quick overview of Claude 3 and we looked at some of the capabilities. I'm going to be building a lot more content on it. I'm specifically interested in the retrieval capabilities of Claude 3 family. This is going to be very interesting because this could be a potential replacement to a Rack pipeline, Even though using Claude 3 or these bigger context models are going to be a lot more expensive. I hope you found this video useful. Thanks for watching and as always see you in the next one.