Can Claude 2 Outperform ChatGPT? Anthropic Thinks So!

Seems to be pretty up to date for this. It's definitely better than ChatGPT. OpenAI just got some real competition. Anthropic, the company behind the Cloud, LLM just released their second version called Claude 2 and it has features similar to GPT4 including code interpreter, but it's absolutely free. You can get access to some of the Premium features that GPT4 offers, but without paying a single penny if you are in UK or US. Now, one great thing about this product is that it has a context window of 100,000 tokens. GPT's 4 context window is 8,000 tokens for the version that is available to public. With this new update, Claude is definitely a real competitor to GPT 3.5 and GPT 4. In this video I will show you how to access the model and play around with it. So Claude 2 has improved performance, longer responses and it can be accessed via API. And they also have this public facing beta website that we're going to look at. I think you still need to pay for API access though. Similar to to OpenAI's models. Now according to them, their models scored 76.5% on multiple choice section of the bar exam, which is around 3% better than the previous version. GPT4 had very similar performance on the bar exam. Now Claude has great reading and writing capabilities, so it scored above the 90th percentile on the GRE Reading and Writing exams. And its abilities when it came to quantitative reasoning are very similar to the median applicant. Again, this is very similar to what GPT4 is able to do. Businesses will be able to use the Cloud2 API and the price is very similar to the previous version. Looking at these performance numbers, this definitely seemed to be a decent alternative to OpenAI's models. However, one thing that really sets it apart from OpenAI's models is this 100,000 tokens context window. This means that you can directly feed it with hundreds of pages of technical documentation or even a book and Claude 2 will be able to process the data. This context windows length is more than three times of the 32k version of GPT4, but that's not publicly available. Okay, before testing this ourselves, let's look at a couple of examples that they have. So it's long inputs, multi step output. So basically here we are looking at a 24 page document and one capability that this one model has that GPT4 doesn't have, that you can upload multiple documents at the same time. So here they're uploading two different papers in the first Part of the prompt they were asking, can you explain the importance of the first included paper describing its exciting new results to me in simple terms? And for the second paper, and for the second paper they are asking for a two column markdown table which highlights details from each section. So basically they are dragging two papers. Right. And as I said, GPT4 can accept only one file at a time using its code interpreter. And then right away it's actually able to generate the responses for both questions. The capability that they are highlighting is its ability to do coding. So it went from 56% in the previous version to 72% on the codecs Human Evolved data set. And it seems like they plan to release more capabilities for Cloud 2. The second example that we're going to be looking at is coding with Cloud adding interactive data to a static map. Basically, they are uploading node JS file, which is going to create this static map. So first they're asking Claude to analyze the original static code. So they're simply uploading the file and then CLAUDE is able to identify the contents of the file and kind of summarize what is actually in the file. Now next they are asking Quad to actually look at the file and add a completely new feature into the existing code. Wow. It was actually able to add the interactive components. Okay, once you sign up, you're going to be presented with a page like this. As you can see, I have been playing around with it, so it kind of keeps track of your conversation very similar to what ChatGPT does. However, the interface is very different from what OpenAI has now. At the top, if you see you can type in your message in here and with that you see this attachment icon. So you can add up to five files and that each can be up to 10 megabytes. That is actually pretty generous given that the code interpreter with GPT4 gives you the ability to upload only one file at a time and you have to pay $20 in order to use that. There are some examples here at the bottom of the page. So let's first check one of these out. So it says, summarize this PDF document in a bullet point outline. So here you see the document and then it actually start generating the response and the speed of generation is actually really good. That's real time. If you have access to GPT4, it does take a while to start generating responses. Now, very similar to ChatGPT or GPT4 and actually included a summary or a title for your chat or conversation. And you can go and rename it if you want. I actually like the interface and its simplicity. Also, if you click on this so it will show you the contents of the documents that you uploaded. And it's here tells you that it extracted 70 lines from this document. Okay, so let's go back and let's create another chat. All right, let's see how good this is at programming and analyzing your existing code basis. Okay, so I'm going to click on this attachment button and here I'm going to upload this in just PY file. So here is what I'm asking it to do. Analyze this code, explain what it does and what potential improvements can be made to this code. So let's see what it comes up with. All right, so it was actually able to correctly identify different components and operations that are happening inside the code. I ran a similar test with GPT4 code interpreter and it gave me similar results. So it's able to identify that this is loading different types of documents. So it gets the overall functionality, functionality of the code. Right. Let's look at some suggested potential improvements. So it's suggesting to use either data set or rate parallelize the operation. That's actually a good suggestion. Could store source documents themselves in document store like elasticsearch to enable keyword search. This is a good suggestion, but not applicable directly to the code base. I think probably it doesn't have the whole context of what exactly is going on because it's just one file from a whole project now ingestion and embedding steps can be split out. That's actually a good suggestion that can be implemented and add more robust error handling. Okay, this is good. All right, so next I asked it to include error handling in the code and it actually did a pretty good job. It updated only the part of the code that needed error handling. So this is pretty neat. And it also give me a summary of what exactly did it do. But this is actually pretty great. Keep in mind that a very similar feature was added to GPT4 just a week ago. And you get that here for absolutely free. Okay, let's see if you can upload multiple times, multiple files at the same time. So I'm going to try to upload these two files. Actually, it's not letting me upload, I guess, that file again. So now we upload one more file. Okay, so I added one more file from the same project and I said, what is the relationship between these two files? And actually did a really impressive job. So it looked at both of the files and then it said, here is the relationship and overall purpose of the project. So it correctly identifies that ingest PYM handles loading documents, embedding them and persisting them to the vector store. Right, so here is different operations that it performs and then it says that the run local GPT then uses the persistent data to run a questioning answering system. And let's say here are the key steps. So load the same embeddings as the ingest py. So ingest PY actually stores the embeddings and the model reads those. So that's just pretty awesome that it's actually able to look at that and understand the relationship. This is extremely impressive given that you are able to upload multiple files at the same time and it has the ability to analyze those files. Okay, I actually just ran into this issue. So it says due to unexpected capacity constants, CLAUDE is unable to respond to your message. Please try again soon or get notified with when paid plans are available. It seems like they are already having some capacity issues, but that is understandable because I think a lot of people are going to be testing this out. Even ChatGPT had some issues in the beginning. Okay, so a couple of things that I actually noticed. You can continue the conversations in a different session and it will actually keep track of the file that you already uploaded. So that is a pretty neat feature. So, so what I did was I asked it can you implement Run local GPT as an API? And it gave me a pretty good skeleton to work with. I definitely need to make some changes, but I think it's a really good start for a model like this. I kind of feel like it's the chatgpt moment for Anthropic. Okay, it seems to be pretty up to date because I asked it who is the owner of Twitter? And the reply was as of July 2023, Elon Musk is the owner of Twitter. So it seems like they just completed training. So this is pretty awesome because probably you're going to get more up to date information from this model compared to something like GPT4 or even ChatGPT if you're not using any type of plugins. All right, let's see if we can actually delete our previous conversations. So I don't see a way of doing it here. But if you come here to the top and then you can click on delete and that will delete your previous conversation. Okay, so I uploaded the Orca paper from Microsoft and I asked it summarize the main findings of this paper and it gave me a pretty nice summary. Now Here is what GPT4 did using the code interpreter for exactly the same paper and same prompt. And this is definitely much better compared to what Claude 2 was able to provide. But overall both of them did a pretty good job. I'm going to be making a more detailed video on comparison between Claude 2 and GPT 3.5 as well as GPT 4. Okay, next, I asked her to create a word cloud for this paper. Initially the speed was pretty awesome, but now it's kind of taking a little bit of time. So here's what it's doing. All right, so, so it gave me a text description. It identified the most important words in the paper. However, it was not able to generate an image when I did the same with GPT4. So using code interpreter, it's actually able to generate this image. So I'm assuming it still doesn't have the ability to generate images in this chat. Okay, let's quickly test its reasoning abilities. So here's a question. Mary and Jeff went to the kitchen, then Jeff went to the park. Where is Mary? Where is Jeff? Right. Okay. So based on the provided information, it goes through a step by step process and then it came up with the correct answer. Mary's in the kitchen, Jeff is in the park. It might seem a very simple task, however, it's not that simple for an LLM. For example, here is what GPT 3.5 things. Right. So it's able to identify that Jeff went to the park, but it's not sure about Maddie's location. Now, I asked the Same question from GPT4 and it's able to correctly say that Maddie is in the kitchen and Jeff is at the park. Here's another one. I actually had to input this twice because it timed out on me for the first time. So the kitchen is north of the hallway. John is hungry, John goes to the kitchen, John grabs the apple there. Daniel is hungry. Where does Daniel go? And why did John go to the kitchen? Now again, this is a multi step deduction that the model has to do. All right, so it came up with a step by step solution. So it says kitchen is north of a hallway. John was hungry. Because John was hungry, he went to the kitchen. Right? And then say, is the in the kitchen? John grabbed an apple. Now, Daniel is also hungry. Now, we don't really explicitly say where Daniel is, but since he's hungry, food is at the kitchen. So he's supposed to go to the kitchen. Correctly identifies, Daniel goes to the kitchen because he's hungry. And that is where the food is right. And again we already know about John. So it also says John went to the kitchen because he was hungry and wanted to get some food. In this case that's the apple. Now GPT 3.5 again has trouble answering this question. Now it's able to deduce that since Daniel is hungry, so it would be reasonable to assume Daniel would also go to the kitchen. But it says however the information doesn't explicitly states where Daniel goes. So I guess like it is kind of correct, but based on the information provided, you would assume that it should say that Daniel went to the kitchen as well, because that's what GPT4 does. So it seems like from this quick test that Claude 2 is definitely computer to ChatGPT. However, I don't know like how good it is compared to GPT4. I'm going to be doing a more comprehensive testing between these two models. This was a very quick look at the Cloth two. Now a few things just to highlight. First, it's publicly available to everybody. It has some features which are similar to code interpreter because it gives you the ability to upload your own documents. Especially you can upload multiple documents at the same time. That's a pretty great feature to have. Even GPT4 doesn't have the ability yet. And the context window of 100,000 tokens like I still have to test it thoroughly to see how good this is, but that in itself is actually a great advancement when it comes to large language models. If you are in the US and uk you will have access to the model. So try it out. See how this is compared to something like ChatGPT. As I said, it's absolutely free to access at the moment. That can change at any time. During my experiments I noticed that it timed out a couple of times so you might run into that issue. But overall the speed of generation is pretty good. These were my quick first impressions. I hope you found this video useful. Thanks for watching and see you in the next one.