New AI Research Proves o1 CANNOT Reason!

So there actually was a new research paper that could put the AI industry in jeopardy. And I'm not saying that lightly, this one is a little bit concerning because it refers to the reliability of these models. So I came across this tweet here on Twitter where it basically says the O1 preview shows a whopping 30% reduction in accuracy when Putnam math problems are slightly variated. So essentially what they're stating is that when they take a popular benchmark and, and they have some slight alterations to said benchmark, there is a whopping 30% reduction in the accuracy of these models when evaluating these models. And this isn't really a good thing because of course, as this guy speaks about Elvis, he basically says that robustness is important as it's key to a model's reliability. Okay. And a model being reliable is something that we need for the AI industry because if a model isn't reliable and it isn't predictable, then we can't have wild scale applications for this thing. Because of course, if you want to use it in finance, you want to use it in business. Business, you're going to want to have to make sure that these models are really, really accurate at what they do. And if they aren't, and if there's so much variance, like a 30% reduction in accuracy from a test result, then that's not going to be something you want to use in house across your different projects. So essentially the paper starts here. You can see it says abstract. As LLMs continue to advance, many existing benchmarks designed to evaluate their reasoning capabilities are becoming saturated and set benchmark saturation is something that the AI industry kind of actually wanted. So this is not surprising. But you can see right here that the benchmarks are becoming saturated. So this is where they're like, okay, so we decided to present the Putnam Axiom original Benchmark, consisting of 236 mathematical problems from the William Lowell Putnam Mathematical Competition, along with detailed step by step solutions. And to preserve this benchmark for validity and mitigate potential data contamination, we, we created the Putnam Axiom validation benchmark with functional variations of 52 problems, basically stating that, look, all we decided to do was alter the problem elements like variables and a constant, and we can generate unlimited, novel, equally challenging problems not found online. So basically that's what they're stating that like, you know, because we're able to just change a few different things about the research papers, we're now able to create tests that the model has, hasn't seen before. And basically it says that we have, you know, See that almost all models have significantly lower accuracy in the variations than the original problems. And our Results reveal that OpenAI's Zero1 preview, the best performing model, achieves a mere 41.95% accuracy on the Putnam Axiom original, but experiences around a 30% reduction in accuracy on the variations dataset when compared to the corresponding original problems. Basically stating that, look, we have 41% on a test that was previously potentially out there in the wild, and then when we change this to a test that we know for a fact that O1 preview hasn't seen before, there is a 30% reduction in accuracy. Now of course, some of you might be saying, why is this O1 preview? Why is this not 01? Why is this not 03 number one? 03? I'm pretty sure they don't have API access just yet because this model is so new. And of course O1 Pro mode at the time or 01 at the time, there was no API when they decided to conduct this study. Now here's the two changes they made. Like I've explained before, the variable change, the simplest variation is a variable change where variable names are altered and the final answer is unvaried variable changes slightly modify the problem from its original statement, which models could have trained on constant change, constant changes modify the numeric properties. So long story short, this is not nonsense, but essentially what we have here is just where they're stating that, look, we just altered the numbers and we just altered some pieces of text and the they actually show us what that looks like. Here you can see the constant change, which is something that is just constant, it just remains stable like these letters. So you can see right here they just put the P's and they change them to the Y's. So you can see right here, the change variables X to W and Y to V and P to L. So that's where a constant value is being changed. Then the variable changes where the values can change. So just numbers changes. They put these numbers. So you can see right here they change to 2011 to 4680. So you can see right here, 2011 was changed to 4680. And this is basically the kind of benchmark that they did. So it's not something like they created an entirely new benchmark. Everything was basically the same, but they just changed the letters and they just changed the numbers. So numbers, numbers being the variables and letters being the constant. So you can see why this is something that when we think about it, the model should get this right because there's only a very small thing that's being changed. And so you can see right here, this is the chart that has, I guess you could say a lot of people freaking out in the AI industry. And I guess you could say those critics who are criticizing these models, stating that they can't reason. I guess this is something that is now on their trump card. So you can see right here it says that the drop in accuracies on this benchmark is statistically significant for nearly all models. And the figure shows the mean accuracies with 95% confidence intervals. And it's pretty crazy is that we can see a large drop here from the original. On 01, we can see that it's around like 50% on the original. And then we can see it drops all the way down to nearly 30% on some cases, or 35%, which of course isn't great if you want to have a model to be really, really robust in the sense that it sees the problem and it reasons through it in the exact same way. And this is something that I talked about in another research paper video, which I will be discussing later. But you can see right here that when we take a look at exactly what's going on, it actually talks about how the low accuracies underscore how difficult the benchmark is, which is of course pretty good. And it does show that the model is able to do pretty decent. So 01 preview is a decent model. But what they state here is that for models like O1 Preview, GPT4O, Claude 3.5 Sonnet and Numen, and Math 7B, non overlapping confidence intervals reveal statistically significant differences indicating artificially inflated performance on original questions due to data contamination. Looking at the numbers highlight significant accuracy declines across all models. GPT4O shows the steepest drop at 44% followed by 01 preview at 30% and GPT4 at 29% and even the very famed claw 3.5 sonnet at 28.5%. So this is a significant drop on these questions when they just managed to change those two things. So we can see here that this is definitely a problem because it's. If we're trying to get these models to be robust, to be reliable, to have wide scale usage. This is an issue that we have to solve because you can't have a model that you test on a specific benchmark. It does really well, and then when you start to use it in production, the accuracy manages to drop to a point where it's no longer at that Reliability threshold where it's viable for a company to use. Now we can see here that I'm going to talk about two things very quickly. One of the things that I want to talk about that could potentially explain this, and this is something that they talk about in the paper is, is of course overfitting. And this is where essentially you create an LLM that matches the training set so closely that the model fails to make new predictions on new data. So this is basically where you can say the model is just trained so much on test data that it performs well on tests, but when it comes to real world scenarios it just doesn't perform well. And this is an issue that a lot of people do say they have with these smaller models. They say that they just overfit them with test style questions like the GSM 8K. And then because of that they, that those small models that we think are getting so amazing, they're only performing well on test style questions, which means they're just overfit for the problem and that's why they can perform so well at that. Now another thing that this paper talks about is they talk about data contamination. So this is where you inadvertently mix samples from evaluation benchmarks into pre training corpora, basically stating that look, sometimes when you're training these models on large pieces of text from the Internet, what tends to happen is unfortunately we can enter a situation so essentially where you have these models that unfortunately, because you're gathering so much data, sometimes some of that test data actually just slips in to the model's training. And of course if it's seen it in training, then that is something where it's going to do well on that test. Which is why when we actually change them and we have those variances and we can actually see whether or not the models do well on those new tests. So this is something that does happen quite a lot in LLMs. And this is why a lot of people can be skeptical on benchmarks. And this is why I personally advise people that even if there is that data contamination, create your own benchmarks on tasks that you have on a day to day basis. And then every time a model comes out, a model gets updated, you put that same prompt into the model and then you'll see if it performs better, if you, if it performs worse. Because everyone has their own specific scenarios, their own specific workflows. And I think while yes, looking at the GSM 8K and those things are very good, a lot of the times we have our own personal use cases that sometimes data contamination might overshadow. So the paper also continues to talk about how, in addition to issues with rigor, GPT4 also displayed logical leaps and incoherent reasoning displayed in Figure 9, where the model simply assumes that an answer is correct and these logical leaps are symptomatic of an issue in GPT4O's chain of thought reasoning as the model prioritizes reaching the final answer rather than providing a rigorous logical output. Now, this isn't something that I would say I'm worried about too much in GPT4O's case, because when we look at the paradigms we're currently in, GPT4O is no longer considered a reasoning model, provided it can reason through basic instructions. When we take a look at what kind of models we're truly using for reasoning capabilities and the raw reasoning abilities, that is the O1 series of models, those are models with vastly more capabilities. And of course if those models are performing underwhelmingly, then that is of course we should be something concerned about because it's like if we develop a model that is a reasoning model and it has incoherent reasoning, then that is of course not good. But nonetheless it shouldn't be the case that we have these models having a large discrepancy between the tests and then of course the unverified tests like these new kind of tests. So it's still definitely not that great because it means that there's clearly some kind of flaws in the models, potential training data where it's been overfitted, there's been data contamination. And so this is where we get to OpenAI's 01 preview. They do talk about out of all models we see that OpenAI01 Preview performed the best on Putnam Axiom Original, receiving 41.9% boxed accuracy, which is 99 out of 236, while other models received less than 20%. And analyzing the answers, we see that most of OpenAI01 Preview responses followed generally the same logical path as the ground truth solution. However, several of these questions contained logical mistakes and inconsistencies. The biggest discrepancy between the model response and the ground truth solution was a general lack of mathematical rigor. Whereas the ground truth solution will make claims to advance its solution and then prove those claims step by step, O1 Preview will often make and use claims without justification. So this is something that once again, it isn't the best kind of thing that you want to see. But then again, this is 01 preview. So I'd have to say that you know, maybe this isn't the best kind of thing, but I would love to see this paper tested again with the actual O1. But still for O1 preview, this isn't the very best thing because claims that justification isn't the best thing and it says thus. 01 Preview shows promised there is still evident flaws in its mathematical reasoning abilities. Now another thing that we do have is we do have Apple's research paper. Like I said before, I referred to this as something that I was previously covering and this was something that I wanted to talk to you about because I wanted to dive into deeper because this is something that I actually remember. When I made that video. I didn't think it was that much of a big deal, but it actually got around half a million views and just goes to show how crazy the AI industry is. And this was basically a paper that was pretty much stating the same thing. And then they talk about the fact that when they change certain things on a paper and they state that we hypothesize that this decline when we change certain things is due to the fact that LLMs are not capable of genuine logical reasoning. Instead they attempt to replicate the reasoning steps observed in their training data. And when we add a single clause that appears relevant to the question, we observe significant drops up to 65%. I'm actually going to include probably a clip from that now because it's just crazy how crazy that paper was. But looking back on it, it was definitely something that pointed us in the right direction. And of course we have this where you can see, right? This was the GSM 8k versus GSM symbolic. Once again, they decided to just kind of create a new benchmark where they just changed these things right here and they saw significant drops across the board in terms of these models. So right here we can see these models. The only thing I can say about this was that the models that really received significant drops was mainly these small models. Which means that potentially I do think that this is the case of smaller models definitely being overfit for the solution. And maybe that is definitely the case here. But to round out this video, it says that while state of the art LLMs have already saturated performance on benchmarks like math, they still struggle with successfully answering questions in Putnam Axiom. To address potential data contamination issues, we introduced Putnam Axiom variations altering the variable names, the constant values or phrasing of the question to create potentially an infinite number of problems not found anywhere on the Internet. And we notice that for most problems, models get significantly worse on the variations than they do the corresponding original questions. And it says our dataset fills the void opened by rapid progress in model reasoning capabilities. Basically stating that look, models have rapid growth in terms of their reasoning capabilities when we look at the future, does this mean that these models are still going to increase their capabilities as the time goes on? And it will be interesting to see how this paper is digested among the wider AI community. And if they do manage to test this again with models like 1 and models like 3, I'd love to see that, because that would of course be a real test, considering that it's one promode those are the true true reasoning models, and exactly how models perform on those tasks.