GPT 4 is Smarter than You Think: Introducing SmartGPT

I have three goals for this video. First, I want to show you a way of using GPT4 to get smarter results. Second, I want to argue that the benchmark results we have for GPT4 do not reflect its full abilities. And third, I want to show you a system that I am developing somewhat cheekily called Smart GPT that is already showing significant results on official benchmarks. It remains to be fully optimized, which I think is exciting in itself. I have shown the system to people at OpenAI who have been quite impressed and I'm going to end with some reflections on where that might leave us for GPT5. But before I get into how it works, I just want to show you one example of it in action to whet your appetite. This example comes from a TED Talk that was released this week.

So suppose I left five clothes to dry out in the sun and it took them five hours to dry completely. How long would it take to dry 30 clothes? GPT4, the newest greatest AI system, says 30 hours. Not good.

On the left you can see GPT4's original answer and it gives this answer pretty consistently whenever you prompt it with the question provided. On the right you can see the final answer from the SmartGPT model which is correct and it consistently gives that answer. I really like how it gives context as well and it provides some of the assumptions that it had in giving this correct answer. Now don't you worry, there will be plenty more examples to go through in this video, including another one from that TED talk. But first I want to give you an overview of what is this Smart GPT model, where did I get my inspiration for it from and how does it work? I'm going to keep it fairly simple because it's the beginning of the video and I know a lot of people won't really care about the inner details that will come later in the video, but the high level overview is this. There are at least three things that have been proven to improve the outputs of GPT4. What's called chain of thought prompting, sometimes called step by step prompting reflection or finding its own errors. And I did an entire video on this called GPT4 can self improve and dialoguing with itself, Entering into a back and forth on its own outputs and deciding which one is best. You can see the title of the papers which contain much more detailed results of course linked above. Now the first paper only came out a few days ago midway through my testing. So my results don't even reflect the full capacity of the model and even if there's nothing else you take from this video, the results from this paper can instantly improve the outputs you get from GPT4. Many of you might remember that prompting GPT4 with let's think step by step improves its results. To give you a very quick reference point, just asking a question to GPT4 gives you 81% accuracy. With that prompt, let's think step by step, it goes up to 86%. But algorithmically, the paper found an improved prompt that can give you even better results. 89% accuracy. All we do, and this is the first part of SmartGPT, is we add answer. Let's work this out in a step by step way to be sure we have the right answer. Now, I have so much to say about why I think this works, but I know many of you won't be that interested in my theories, so I'm going to save them to the end for those who are interested. Some of you just want the results, so I'm going to get to those first. So far you might be thinking, well, thanks, Philip. That's a cool prompt. I'm going to use that. But what's this whole smart GPT about? Is it just a single prompt? No. I believe with evidence there are ways of leveraging even better results than just using a great chain of thought prompt. So let's move on to the next part of the system. These different outputs in the middle. For my tests I typically did three outputs, but of course, depending on the context window, it could be far more than that. And I'm going to talk about ways I could further improve this model. Or we could later on in the video, just to restate these outputs are when you take the user input and add the word question at the start and then at the end add answer. Let's work this out in a step by step way to make sure we have the right answer. And at this moment many of you are thinking, what is the point of multiple outputs? It's GPT4. It's just going to give you the answer it thinks is best and that's it. Well, actually, it doesn't quite work like that. These models have a temperature between 0, 0 and 1. I believe the default for GPT4 might be around 0.5. And simplifying massively, this determines how creative or conservative the model is in giving its outputs. So given that GPT4 tries to be fairly creative, you don't get the same output every time. The output is randomly sampled according to an internal probability distribution. So you can get situations And I face this hundreds of times where some of the outputs are correct and others are incorrect. And this is where reflection comes in sometimes, definitely not always, but sometimes, quite often. GPT4 can detect the errors in its own output. And many of you will notice at this point that the prompt that I used to elicit GPT4 to spot its own errors contains the same step by step prompt I used earlier that has been shown to produce good results. So to summarize, sometimes at this stage GPT4 detects the errors that some of its outputs have made. Definitely not always. There are certain questions it just simply can't spot the error. But sometimes it can. And then I get it to engage in a dialogue using a format similar to one in this paper published last month. It's a short dialogue, and this is the step I believe that can be most optimized in the future. I envision an entire council of advisors made up of GPT4 imitating mathematicians, judges, etc. At the moment it's just being a resolver and printing a final improved output. Anyway, I'm going to get back to the theory later in the video because I know some of you will be getting bored at this stage and want to see more practical examples and the results from my benchmark tests as I don't have the GPT4 API key. Yes, I had to manually input each of these steps hundreds of times, waiting sometimes three hours between each go because you can only do 25 messages every three hours. On the left you can see the three outputs when you ask it to think step by step. And then you have the researcher step in the middle and at the top right and finally the resolver step. Notice here I was using the original let's think step by step because the paper hadn't yet been published on improving that prompt. But it's time for the second example from that TED talk. And then I definitely will get onto the benchmarks a different one.

I have 12 liter jug and 6 liter jug and I want to measure 6 liters. How do I do it? Just use the 6 liter jug, right? GPT4 spits out some very elaborate nonsense.

Of course I tested SmartGPT with that question and you can see the difference between the original GPT4 which gives this incredibly convoluted bad answer, and SmartGPT the final answer output. Now, at this point I know many of you will be impressed, but you'll be thinking I don't have time to input things five times. Well, I'm developing a model where it can all be done automatically. Here is a preview of how it works, but of course at the moment it has to use GPT 3.5 Turbo because I don't have the API key of GPT 4. But the EPIC thing is this. You just ask a single question I've written Ask SmartGPT a question and of course it does take a little bit longer to respond because it's doing five or six calls via API, but it does output the final answer from the resolver step. I will be honest and say that GPT 3.5 isn't as good at reflecting or resolving, but this is an example of a question where the original ChatGPT consistently gets it wrong and SmartGPT 3.5 gets it right using this program. Remember, all you have to do as a user is type in a question as normal and it goes through this entire five or six step process behind the scenes. By the way, this was a question from mmlu, which is a famous benchmark which I'll get to in a second. Here's one last practical example before I get to that benchmark. I know many teachers use ChatGPT and GPT4 to create quizzes for their classes, and here is the same question put through GPT4 and SmartGPT. The question is Create a high school algebra quiz with five questions and answers and explanations at the end points for spotting the difference. But if the teacher had handed out the original quiz, look at the answers for question five. It says the answers are 1 and 1.5, but then in the explanation it gives the final answers which are correct by the way of 3 and 0.5. So that would really confuse some students. At the reflection stage, SmartGPT spotted that error and resolved it. And as you can see, the answer for question five has the correct answers straight away. If at any point you're wondering if I completed the OpenAI ChatGPT prompt engineering course, the answer is yes. But it didn't inform too much of my thinking. It was more for beginners and I had already factored in things like giving the model time to think and writing clear instructions. The benchmark that I chose to test SmartGPT on was the famous MMLU Massive Multitask Language Understanding benchmark. As you can see, the state of the art is indeed GPT4 with 86.4% accuracy. And you know, OpenAI think it's a big deal because it's the benchmark mentioned on the front page of their technical report report. Without boring you too much, I extracted the questions from the test set of the MMLU data file. And I didn't pick the topics at random. I went for those that I thought GPT4 would find the hardest. Delving into the original MMLU paper, you can see that GPT3 found formal logic the hardest, scoring just over 25%, which is random chance. It's a four question multiple choice test, so around 25 or 30% is pretty bad. And notice they helped out GPT3 here. They did it few shot, meaning they gave it five successful examples before asking it a new question. It's the same thing they did with GPT4. They did it 5 shot. But just before I show you the results, there are three things I want to mention here. First, I was curious how smart GPT would do without any help. Zero shot. Second, I wanted to do it zero shot because people using GPT4 don't typically give five successful examples before asking GPT4 a question. They just want code or a quiz or a poem or an example. They don't often provide five brilliant examples of code before asking their question. And third, if I can prove it works zero shot, then of course future refinements can be made to push the results even further. And here are the results from the first 25 questions from the formal logic test set of the MMLU. I did many more tests after this, but you can see from this set if you just ask the question, you get a lower overall accuracy. But of course, course 68% for GPT4 is still a huge improvement over GPT3's around 25%. What happens when you add. Let's think step by step, which as we know now isn't the fully optimized chain of thought prompt. Well, on average you get around 74, 75%. That was 75 examples inputted manually. And I still have all the tabs open. I'm keeping them open because I'm compiling a spreadsheet with the actual output. But what did the resolver get? Drawing upon GPT4's ability to reflect and engage in dialogue with itself, here's Lennart Heim, an AI governance researcher, suggesting a score of 95% on the MMLU would be reflective of AGI like abilities.

I do think I have like a 50% chance like within the next 20 years or so there might be something what we might call an AGI or transformative AI. What do I mean by this? Well, maybe can measure it on benchmarks. There's like this famous MMLU benchmarks. Like yeah, there's something which like score is like 95% on this.

Going back to the results, if a smart GPT like system can automatically resolve half of the errors that GPT4 makes on the MMLU, that would increase its score from around 86.4% to around 93%, which is not far off 95%. Remember, his prediction was a 50% chance in 20 years. I'm talking about GPT4 now, for those who are still skeptical, I'm going to show you plenty more results now and then walk through the papers that give the theory as to why this works. One thing that I forgot to mention earlier is that the human expert level on The MMLU is 89.8% and that's taking the 95th percentile of human test takers. And remember, those are domain experts in each of these subtopics. What we're doing is testing GPT4 or Smart GPT on all of the topics simultaneously. So even if smart GPT like systems can't quite reach 95% and I think honestly they'll get pretty close with all the refine that I'm going to suggest, I think they should almost certainly be 89.8%, which is the human expert test taker level. Intrigued by these results, I then put it through the college math test from the mmlu. And remember, this was before using the optimized version of the step by step prompt. Obviously I'm not going to go through all the questions here, but let's skip to the final results. We have zero shot accuracy 6 out of 15, which is 40%. The average when you add let's think step by step was 53.5%. And then the final output of the resolver model had a 60% accuracy, so it couldn't quite resolve half of the errors, but the overall pattern held up. In case anyone is wondering about methodology, I kept the formatting identical for every question. I always opened a new tab for each question. It wasn't looking at the context of what it had already put out. Each attempt was fresh aside from the resolver model which looked at the context of the researcher's output. And again, as you can see from example 40, it wasn't like the researcher could always spot the errors or that the resolver could always pick the right option. Sometimes the let's think step by step prompt gave the right output, but the resolver couldn't quite distinguish it. The optimized prompt gets a slightly better output and upon reflection the researcher can sometimes but not always, spot the errors of those outputs. And sometimes, but not always, the resolver can spot based on those flaws, which answer is best. These are incremental improvements. Sometimes GPT4 simply can't get it right. I have noticed a few themes in those questions. Anytime it comes to division, multiplication, characters or counting in general, GPT4 tends to make mistakes that neither the researcher nor resolver can spot. Of course, integrating a few tools via API would likely solve those issues and I don't want to preempt the conclusion too much, but I believe a smart GPT like system them with tools integrated could probably score around 95% right now on the MMLU, especially if it was helped out with few shot prompting to add weight to that preliminary conclusion. I tested it on certain topics and had to stop because it simply got the questions right every single time. For example High school psychology from the mmlu. I then tried prehistory, which it also aced before finding machine learning where I got more interesting results. Zooming in this time the raw score was 65%, the chain of thought let's think step by step average was 71.6% and the resolver model got 80%. Let's now look a little deeper into why all of these steps might improve the end result. In reply to the original let's think step by step paper which was published around a year ago, Andrej Karpathy said this adding something like let's think step by step to the prompt is a way of using the input space for computation that you'd normally want in the hidden state of the model. Instead of the workings out being done in the activations of the neural network, it's done in the discrete tokens of that input space. And he adds, did not supersee this coming. And here is the paper released three days ago that improves upon that original prompt. They also did their testing zero shot like me and they tested many prompts starting like I did with just direct prompting, just asking the question like 99% of users would do of GPT4. And then they tried like me, the well established let's think step by step prompt. They also iteratively tested seven original prompts as well as the prompt that I've now integrated into SmartGPT, the Let's Work this out in a step by step way, etc. They share my opinion that zero shot prompting setups have the benefit of not requiring such task dependent selection of exemplars. You don't have to find correct examples, it just does it all for you. Here are the end results for GPT4 that we saw earlier showing the difference between asking directly your question and using these refined prompts. Notice that this technique is somewhat model dependent, and it doesn't have the same effect on smaller or weaker models. Before we move on to the next paper, there is one somewhat failed prompt that I want to pick up on. It's this self critique prompt where they ask answer the question, then critique the answer based on the critique, reconsider the other answer options and give a single final answer. And you might wonder why didn't that prompt perform best when we know that reflection and dialogue can work? My theory is because it's trying to do all of it in one prompt. Through my hundreds of experiments, I've noticed that GPT4 can only handle so much in one go. It simply gets overwhelmed.