Dr. Zsolnai reviews Anthropic's Claude 3, which comes in three versions and can read images and process 200k tokens at once. Claude 3 Opus beats GPT-4 on several benchmarks, including GPQA, and costs less to run. The speaker notes that different prompting approaches and GPT-4 versions make direct comparisons tricky.
Haiku, Sonnet, Opus
200,000 tokens
159 countries
2024-03-07
Oh my. We just talked about Claude 2, anthropic's intelligent AI assistant. And what is this? Are you kidding me? Claude 3 is already here? And what are you seeing? What I am seeing? It claims to have beaten the mighty GPT4 on all of these tests. So is this the smartest AI assistant ever? Wow. Now at the end of the video we will take a closer look at this together, because it gets a little more complex than that. Dear fellow scholars, this is two minute papers with Dr. Zsolnai and I have to say there is a lot to like here. It comes in three Haiku, Sonnet and Opus. It is multimodal, so it takes images and books too. And more. For instance, we can search for something on the Internet and it gives us a different plot with these numbers. But why? Now wait a second. So it just read the numbers and copied them? Nope. No sir. It looked at this image, wrote a piece of code that would recreate this plot, plus added some useful context to what happened, when and why. Ok, that's nice, but how close is that to the actual data that was shown? Wow, it is indeed close. Good job, little AI. And it can not only read and recreate all this historical data, but it can also try to simulate it into the future. And it can rerun the same kind of analysis for several different countries at the same time. We can also get a pie chart that shows what it expects the world economy to look like in 2030. This would take a paid intern, hours and hours of work, and it can just do it in a few seconds. And we get a written report too. A full multimodal analysis. Really nice. The context window is also 200k tokens long. Okay, but what does that even mean? Now we had a look at this before with Claude 2. This context window means how much information it can read through and remember. For instance, we could give it a book or a sizable PDF file. It reads it almost instantly and it could summarize all this data for us. This is what a context window means. However, the the context window shouldn't only be measured in number of tokens, but in quality as well. You see, version two had a really wide window, but occasionally it forgot some things from earlier. And what about version 3? Now hold on to your papers, fellow scholars. And would you look at that? It remembers. Fantastic. Now benchmarks. This is huge. They say that Opus, their biggest network, scores better than GPT4. The king itself on pretty much all of these tests and even the small one, Haiku, is showing very respectable results. And look, it is going to be 10 to 60 times cheaper compared to the smaller models. This is an extremely important aspect. The true age of AI is coming when we can do all this for nearly free. And we are getting closer and closer. We will have AI teachers for our children who know everything, can teach in any style, who are there 24, 7 and never run out of patience. And all this nearly for free. What a time to be alive. And this, this really caught my eye. Gpqa this is an incredible result. This data set has questions that make specialist PhD students in organic chemistry, molecular biology and physics blush. And they say that it beats GPT4 even there, and not just a little. Now we are experienced fellow scholars here and to temper our expectations they there are at the very least three caveats we should consider. One, the prompting techniques are not the same everywhere, but they seem to be in cases a little stricter against Claude that is good news. Two, these tests are not everything. We have to be careful about data leakage, which means that some of these questions and answers may have leaked out to the Internet. After all, anyone can upload them anywhere and and this would reduce the validity of these results. Now note that we had just shown a few questions, but not the correct answers. These are in the paper too, so I hope that's okay. And three, there are independent benchmarks too which temper the expectations a bit. Also note that there are many versions of GPT4 which vary quite a bit and they have tested against some version of GPT4, but not necessarily the newest version. Perhaps that's why the independent benchmarks look different. So that is an important detail. But at this point I think it is fair to say that it can kind of keep up with GPT4 which is a huge achievement and it is available in 159 countries. It seems to me that you can even try it out for free. The link is available in the video description. Now the real test is always the performance in practice in the particular areas you are interested in. For instance when asking it to convert this image into front end code. So create a website with this design if you will. Well there's that. And ChatGPT. Oh yes, it just keeps doing the work. So when talking about an AI assistant there are so many variables to consider. Try everything you can get your hands on and choose the one you like best. That is the way of the scholar. Also get this. Around mid April I will come to San Francisco and the US for the first time ever. I will stay for about a week and speak to you fellow scholars at this conference. Oh my. Scholarly Content in Person Register if you're interested. The link is in the video description and I would like to greet and talk to as many of you fellow scholars as possible. Note that I think the seats are limited. Last time we did something like this was in London and there were so many of you fellow scholars there we couldn't even see the end of the line. And what did you come for? Of course, the papers. Can't wait to do it again. I'll bring some presents to you this time too. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.
