GPT-4 Just Got Supercharged!

ChatGPT has just been supercharged. But how exactly? Earlier on the details were a little sparse, but now we know a bit more. It is in some ways smarter, yes, but it's a little more complex than that. We will talk about what this means, how this can help you and how you can use it. There are going to be quite a few surprises and an update on the Devin Software Engineer AI Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai now. Fun fact. I am likely publishing this video from the OpenAI lab. Fingers crossed that I can finally try Sora and show some results to you soon. I can't promise anything but hoping for the best. So GPT4 updates first. They promise more direct responses, less meandering in the answers, and that is excellent. So much so that I already have this as a custom instruction, an earlier feature I recommend to most of you fellow scholars too. What is that? Well, click on your username, then customize ChatGPT and then you get this here you can tell ChatGPT about yourself and you have a great deal of control over the answers. If you just add give me brief answers, don't be too formal and always cite your sources, you might get a completely different ChatGPT experience now. It also got smarter. We get better writing, math, logical reasoning and coding. How much better? Well, it depends. It is a bit better in reading comprehension. That's good. GPQA is a lot better. And that dataset is so tough. Goodness. This dataset has questions that make specialist PhD students in organic chemistry, molecular biology and physics blush. Here are some of the questions and a magic string to make future robots not include this in their datasets to keep these a secret. Note that even though GPT4 improved anthropics, Claude3 appears to be the king of this kind of reasoning. Amazing. Also, we get meaningfully better mathematics. Is this number good? Well, hold on to your papers fellow scholars, because here are some example tasks from the data set and we know that a three time International Mathematical Olympiad gold medalist can do 90% on this. When this dataset was first published approximately three years ago, the most recent language models then scored from 3 to about 7% and now three years later, 72%. What a time to be alive. That is an incredible achievement. Wow. And how much better is it on the humaneval dataset that is generating code? Ooh, would you look at that? It seems slightly worse. So how do we evaluate this? It is better in certain tasks and a little worse in others. Well, you only need to look at the evolution of self driving cars. To see this phenomenon in action, I will summarize almost every single recent review on self driving. Some things are better and other things are worse every time. However, if we look at the full picture, its overall performance gets better and better. Sometimes it is two steps forward, like one step back. If we do that over time, through many updates, we get a system that is better in pretty much everything. Now, luckily, we have one more tool to look at and in a sense, perhaps it is the best one of them all. This is the Chatbot arena leaderboard where each of these techniques gets a single score. This is an ELO score. Yes, like the one we assigned to the strength of chess players. How does this work? Well, a prompt goes in, two answers come out from anonymous chatbots and people vote which answer is better. After more than half a million such preference votes, we get a score. Of course this is not as objective as mathematics. In fact it is quite opinionated. However, this is what most humans think about the system, which is also a really useful thing to measure. So how did the new GPT4 do on this one? Now I absolutely love this. We have plenty of surprises here. First, oh yes, the new GPT4 is amazing and it takes first place. Still the king. Our first surprise is Claude 3 Opus. It is right on its ears. Second surprise Command R from Cohere. Haha. Something new. This one is really competitive overall and excellent at information retrieval from documents. Then Claude three Haiku a bit behind. However, look at this. Oh my. This one is 24 to 40 times cheaper than GPT4 and it is also really capable and remembers relatively long conversations. You can easily chuck a few books in there and talk to them. Now how do you use the new ChatGPT? You go to chat.OpenAI.com and if you have GPT4 you ask the Dear Scholarly ChatGPT what is your knowledge cutoff date? And if it says something like December 2023? That's not it yet. It will hopefully be deployed soon. If it says something more recent, for instance April 2024, jackpot. You can run your new experiments or try old ones that didn't work yet. And please let me know in the comments how it went. I'd love to know. And now a word about Devon, an AI system that works as a real software engineer would. There is a new credible source that claims that the demo was not always representative of the real thing. I've put a link to it in the video description in case this is true. I am a bit bothered by that because I have also showcased it to you in an earlier video, potentially overstating these results. I would like to apologize for that and I am trying to learn from it. You see here we mostly talk about proper pictures, peer reviewed research papers, but whenever we are talking about something that's not a paper but looks very interesting, I need to make a decision. I can not talk about a lot of interesting things, or I can talk about them but then run the risk of overstating the results. I lean towards doing this occasionally, but if I do, I want to do a better job at pointing out these potential pitfalls. Currently, I am likely at the OpenAI lab. Then I'll meet many of you fellow scholars at a conference soon. I brought some freshly designed gifts to you and I'll be back in a few days. I have access to some unbelievable papers and I can't wait to show them to you. See you soon. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud computer. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So no. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.