SmartGPT: Major Benchmark Broken - 89.0% on MMLU + Exam's Many Errors

Since late April, myself and machine learning engineer Josh Stapleton have evaluated over 120,000 answers from GPT models to explore their limits. In my original SmartGPT video, I showed that even popular TED talks calling GPT4 stupid were not accurately testing what GPT4 could do. And actually, it could easily get such questions right. Little did we foresee that come the summer, our test with GPT4 would be revealing a host of mistakes in an official, globally used benchmark, uncovering concerns that even OpenAI and Google don't appear to be aware of. But by the end of the video, I want to show how you can tangibly benefit from our experiments, including in unexpected domains like medicine. Where to start? Well, here's a super quick intro to those of you who haven't seen the original video. Smart GPT was a way of using the latest prompt engineering research to trigger better performance in a model like GPT4. Getting the model to think a bit, aka use some tokens before giving a final answer was key. Another important element I talked about in that video was the power of getting the model to self reflect, an insight I drew on from talks with the lead author of the famous reflection paper. My manual experiments showed that using optimized prompts, reflection and self dialogue, you could boost performance in almost any task, and I demoed the improvement on formal logic and college mathematics. But there was a problem. Which is why you guys haven't heard about SmartGPT in a while. How could I systematically benchmark GPT4 using these methods when I'm just one guy? Well, enter Josh Stapleton, machine learning engineer extraordinaire. Without him, it would have been impossible to build out such a fleshed out flexible code base with which we could systematize experiments and iterate rapidly. But then we both quickly realized that there was another problem with benchmarking the original version of SmartGPT on tens of thousands of official questions. It would be hell to manually extract out the final answers within pages of reflection and resolving, not to mention cost tens of thousands thousands of dollars. And trust me, a month of YouTube advertising would not even cover the first hour of that run. Unfortunately. And no, we would never compromise by asking GPT4 to grade its own answers. It would be unscientific and inaccurate. The infamous MIT paper is enough evidence of that. GPT4 didn't get 100% on an MIT degree and this paper was withdrawn. So yes, we had to lower the power level of Smart GPT, get rid of the reflection and resolving deliberately sacrificing some of intelligence because we simply couldn't afford to unleash it fully. And yet we still got a new, albeit unofficial, record of 88.4% on the MMLU. That not only beats the 86.4% recorded by OpenAI, it beats the projections for 2024 that Metaculus recorded before ChatGPT came out. And yet we are both convinced that there are at least a dozen more ways performance can be further boosted using existing models. Yes, that might mean GPT4 getting result reserved for June of 2025. The thing is, we have hit the limits of what a self funding team of two can do. Before I briefly touch on what the MMLU is, I am happy to say that all of our Results and answers that's 2,850 for the GPT4 run and over 120,000 for GPT 3.5 are freely available to browse in a GitHub repository linked in the description so what the hell is MMLU anyway? Well, it is arguably the best known benchmark of Langu language model performance. It stands for Massive Multitask Language Understanding Massive because It has over 14,000 questions and multitask because it covers 57 different domains. The idea behind it was truly fantastic and it is important enough to feature prominently on the first page of the GPT4 Technical Report. In the past I have said that getting 100% on this test would be a good sign of AGI. Others have talked about 95%.

I do think I have like a 50 chance like within the next 20 years or so there might be something what we might call an AGI or transformative AI. What do I mean by this? Well maybe can measure it on benchmarks. There's like this famous MMLU benchmarks like yeah there's something which like scores like 95% on this and the paper itself.

Notes that an 89.8% performance represents human expert ability which as you can tell from the title we are achingly close to beating. And as you'll see In a moment, GPT4 with the full power of prompt engineering could likely get upwards of 90 to 92% right now. And frankly, whether it's GPT5 or Gemini, that 95% threshold should easily be broken by next year, not in 20 years. If we didn't use the full power of SmartGPT, how did we get 88.4%? And why does the title say 89%? Well, let me show you the two facets of SmartGPT that we did use. The thing is, the MMLU demands a Single character answer A, B, C or D, and that answer must be immediate. Now imagine taking a test and the very first thought you had had to be your final answer. On a quick tangent, this is believed to be a key reason for hallucinations. This was a great paper on how language model hallucinations can snowball from the first few tokens. As they say, the language model first commits to an answer that's before outputting the explanation. And this is a problem because transformers cannot find the answer within one time step because of their limited reasoning abilities within that time step. And why don't language models like GPT4 and change halfway through? Well, they prioritize fluency and coherence at the expense of factuality. But rushing an answer in your first token is particularly hobbling in questions like this requiring deeper thought or calculation. It's fine for questions that need memorized knowledge, but not for questions like these. We went through all of these subjects in the MMLU and characterized around a third of them as requiring that kind of deeper thought. And of course most ways that you use GPT4 also require some thought, but I'll touch more on practical uses at the end. Open source teams and groups like OpenAI and Google all draw on the dev set when testing a model. Notice the five questions, each with a single character answer. We were not of course, the first team to realize that this underplays the abilities of the model. The Minerva paper from Google said the standard way of evaluating on the MMLU is to construct a five shot prompt prompt out of the dev set. So what they did instead, like us, was to use a prompt which has a chain of thought before outputting the final answer. You can see some examples below. Essentially it allows the model to think a bit first and gives it a scratch pad. There are other theories though, like the length and detail of the exemplars triggering different weights of the model. This paper from two months ago used longer exemplars for the model Moral scenario subject within the mmlu. With these five custom exemplars plus self consistency, which I'll get to in a moment, they saw accuracy go up to 80% from 68%. But before even this paper came out, Josh and I were sourcing and crafting bespoke exemplars or the 21 subjects that we deemed would need the most working out. But the other subjects we may do with the normal dev examples, OpenAI and Google don't do this for their benchmarking, underplaying the abilities of their model. So why doesn't everyone do it? You might ask. My theory is that it's because you have to hand grade every answer rather than just lazily check for an exact character match through auto grading. Essentially you're taking the time to listen, which is the least that we can do, I feel. As we approach human level intelligences, even though it still took weeks to make checking easier, we taught GPT 3.5 and GPT 4 through our exemplars to always end with a final answer in the same format. Lesson one, therefore for everyone watching, is make the first token the final answer for the question you ask. Lesson two comes from a paper on self consistency, which in a nutshell says that taking the highest probability answer, sometimes called greedy decoding, doesn't always reflect the best answer the model is capable of. In other words, don't take the model's first answer as its final answer. Take this example, the highest single probability answer was this and that was incorrect. For OpenAI, it would now be over. It's incorrect done. But sometimes if you look at all the different answers that a model might give and then take the majority answer, the final answer that came up the most often, it can get it right. Interestingly, in the Minerva paper they used 256 samples, although only 16 for the MMLU. OpenAI even put a little footnote in their GPT4 technical paper, admitting that they don't use that approach. But Google does. And yes, this can significantly affect the final results. Look at the boost going up to 40 sampling paths and it hasn't fully levelled off yet. These aren't redos where you keep trying until you get it right. This is letting the model explore its full probability distribution of outputs and taking the truly most probable final answer, letting the model think, not rushing it. For our runs we limited ourselves to nine samples and took the majority vote. But of course the results could have been dramatically better if we did 40 samples or indeed 256. Now aside from these two hard won lessons, which I'm going to show how all of you can benefit from, the other difference from the previous state of the art 86.4% was that we did use the most current versions of each model, so the models may have independently gotten better or worse in certain topics. But I would say that our broad findings do run counter to any simple it's got dumber narrative. And if I had to guess behind the scenes, OpenAI have implemented some fine tuning involving solving step by step solutions as I see that phrase cropping up in answers where it didn't used to, and that particular trick from the original Smart GPT seems less effective than before. I'm now going to ask Josh to talk about our state of the art score not only with GPT4, but also with GPT 3.5. But just before I do, here is a hint of why the title talks about breaking a benchmark. I'll show you how GPT4 itself encouraged us to question many of the answers in the mmlu, starting with one strange answer and then like pulling on a thread, leading to the discovery of at least 80 and likely hundreds of errors in the test, enough to significantly affect final results by up to 2%. And given that the differences in say, the open Source language model leaderboard come down to as little as 0.1 of a percent, that's pretty big. Yes, we've been in contact with some of the authors of the test over the past month to check our findings and I'll say more in a bit bit, but first, here is ML engineer Josh detailing how the magic happened. Josh, by the way, is a precocious AI consultant working on a Master's at Imperial College London.

Hi everyone, nice to meet you all. My name is Josh Stapleton and let me show you the version of SmartGPT we used. The Smart GPT framework is highly parallelized and can handle industry scale use cases. We used a thread and a syncio based approach to make simultaneous calls to the API at answer, option, answer and subject levels, stacking parallelization upon parallelization. This led to crazy iteration speed boosts. For example, we were able to complete the final GPT4 run in under two hours. Generating single answer options in series would have taken weeks. We did two large runs using smart GPT, first with GPT 3.5 and then GPT 4. The 3.5 run was on the entirety of MMLU for a total of 9 times 14,042 questions, 126,000 answers. This was a mammoth effort to manually grade, but the Smart GPT innovation and hard work ended up boosting GPT 3.5's performance by a significant 3.7% from 70% to 73.7%. The GPT 4 run using SmartGPT also beat the OpenAI MMLU benchmark score substantially, and this run actually resulted in the discovery of a number of problematic MMLU questions, which Philip will talk about shortly. The cost to run GPT4 on all MMLU would have been too high for us to self fund, having already each invested four figure sums. So we used a representative subset of 2,850 questions from the total of 14,042. Of course, fully weighted to official standards, SmartGPT is a model agnostic, parameterized and highly flexible system that can be applied to disparate use cases. We are already working on applications in a number of domains in both the public and private sectors. The system is evolving and improving constantly under the hood as we continue to innovate. While the current system can get state of the art results with the ability to handle enterprise scale data, there are a number of known ways to improve it which we aim to implement in the near future. From better and more numerous automatically sourced exemplars to LLM driven prompt optimization to fine tuning, we are just getting started with Smart GPT and we are uniquely positioned as a tiny team to integrate both our own ongoing improvements as well as promising discoveries in the field as they arise. Back to Phil.