OpenAI FIRES BACK At Leakers On GPT-5s Performance

So Sam Altman actually just fired back at those who have leaked information about GPT5 Orion. In this video, I'll dive into everything you need to know and why this is important for the future of AI. So it actually all starts with this Bombshell article. The article states that OpenAI, Google and Anthropic are struggling to build more advanced AI. I'm going to be explaining to you exactly what this means, but basically this article states that three of the leading AI companies are seeing diminishing returns from the costly efforts to develop newer models. And here's specifically what it states that has Sam Altman firing back. It says that GPT5 is essentially a disappointment. It said that the model known internally as Orion, which some would call GPT5 as they have changed the naming sequence. It says that this model did not hit the company's desired performance, according to. According to two people familiar with the matter who spoke on condition of anonymity to discuss company matters. And as of late summer, Orion fell short when trying to answer coding questions that it hadn't been trained on. So they're basically stating that, look, this model, the model that is supposed to be the successor to GPT4O or as many know it, to GPT4, this model isn't living up to internal expectations, which is quite disappointing, disappointing considering the hype around OpenAI's next generation. But there's more to get into. It says that Orion is still not at the level OpenAI would want to release it to users, and the company is unlikely to roll out the system until early next year, one person said. And of course the information also reported on this article where I spoke about how GPT5 is experiencing a slowdown. And you can see from this that they're basically stating that, look, this model that was meant to come out later this year, which is supposed to be GPT5, isn't currently performing at the rate that we thought it would. And because of this, it now means that this model is going to come out even later. Now, a lot of this information is actually rather fascinating because some individuals on the Internet in the AI community have been predicting this for quite a while. And this is where we get into the drama with Sam Altman. So remember how, okay, right now in this current space, this, it seems as if AI is slowing down, at least with the GPT series. Prior to this, there was Gary Marcus, who actually said this in an article. He stated that deep learning is hitting a wall. What would it take for AI to make real progress? And I'm going To summarize this article really quickly, but basically in this piece, Gary Marcus critiques the limitations of deep learning, and he argues that while AI excels in pattern recognition tasks like image and speech processing, it falls short in areas requiring real reasoning, common sense and understanding. And he basically says that deep learning models often function as black boxes, lacking transparency and interpretability, which raises concerns about their reliability in critical applications such as radiology, autonomous vehicles, and other real things that we're going to need in real life. And one of the takeaways I need you guys to take away from this, because we're going to come back to this point later, is the fact that he advocates for a more integrated approach to AI, combining deep learning with symbolic reasoning to address these shortcomings. And his perspective is not one that is really popular, but he's been very vocal about this for quite some time, especially since March 2022. Now remember the statement, okay, deep learning is hitting a wall. That's essentially a statement that had been echoed in the AI community for quite some time. And every time AI makes an advancement, people have literally been using this and say, haha, deep learning is hitting a wall. And then showing astonishing new benchmarks. Now of course, with all of this talk in the space with articles saying that, look, these companies are struggling to build advanced AI, GPT5 is apparently at a disappointment. And of course referencing this article from 2022 which states that deep learning is hitting a wall, Sam Altman actually tweeted yesterday, or I think perhaps today, that there is no wall. Of course, like I said before, this is referring to the article released two years ago basically stating that deep learning is hitting a wall. Now, Sam Altman isn't the only person that stated this on Twitter. Will Depoo, who is actually working on Sora AGI at OpenAI, also stated something rather fascinating. He stated that scaling has hit a wall that is 100% evaluation saturation. If you don't know what this means, he's basically saying that the only war that AI is going to hit is one where all of the current evaluation methods are going to be completely blown out of the water, essentially meaning that, look, these benchmarks that we're training our models on are going to be completely saturated by the time 2025 is in full swing and we have future iterations of certain models. Now Will the POOH isn't the only person who has said that benchmarks are going to be saturated. If we do take a look at what Sam Altman said in the OpenAI, Ask me anything. One of the bold predictions for 2025 was that apparently OpenAI are going to be able to saturate all the benchmarks, which is a pretty bold claim considering how difficult a lot of these benchmarks seem to be and for the fact that a lot of these benchmarks are supposed to stand the test of time. Now remember guys, the previous statements, deep learning is hitting a wall and news articles are basically stating that, look, these companies are struggling to get advanced models. So quickly. What I want to do, guys, is show you guys something that Gary Marcus recently said, which was, folks, it's game over. I won. GPT is hitting a period of demolishing returns like I said it would. Now, honestly, credit where credit's due because Gary Marquez did state that by the end of 2024 we will see seven 10 GPT4 level models, no massive advance or either no GPT5 or disappointing GPT5 price wars, very little moat for anyone, no robust solution to hallucinations, modest lasting corporate adoption and modest profits split 7 to 10 ways. I have to be honest, as someone that's been in the AR community for quite some time, this is something that I have witnessed on a scale that you wouldn't believe. There are 7 to 10 GPT4 models. There hasn't been any massive advance in the GPT series. Don't worry guys, I still know that 01 exists. And of course recently there has been no GPT5 or disappointing GPT5 according to these leakers, people who have spoken to these news reporters. Now, of course there have been price wars and of course one of the crazy things is that there is no robust solution to hallucinations. Even In a recent AMA OpenAI literally said that hallucinations are a huge problem that they are still working on. Now remember, this is all about the GPT series, okay? Because there is a lot of information about the 01 series that I think a lot of people are ignoring. And this is where Sam Altman is sitting back and where you actually need to pay attention. So pay attention to this part of the video because this is where things start to get crazy. So someone on Twitter actually asked Sam Altman and they basically said what about Chollet's Ark? Eval and Chollet's Ark evaluation is basically the toughest evaluation there is currently for LLMs. Of course there are a variety of different LLM benchmarks and evaluations, but this one is the toughest because this kind of question, the way that they've set it up and is they set it up so that these LLMs they literally cannot memorize exactly what's going on. And the reason they've done it that way is because they want to test how an LLM truly reasons about a problem that it's never seen before. A common critique of large language models is that since you're packing all of this data into it, you're essentially just fitting the model with the data it needs to answer those questions. And essentially you're just building a system that can retrieve the answers at test time, and it's not actually reasoning about the solutions, which is why this evaluation was built to test how they reason on problems that they could have never studied before. Now, in regards to this evaluation, which you can see on screen, those are examples of how you'd solve the problem. Sam Altman actually said that in your heart, do you believe that we've solved that one or no? And in my heart, I do believe that they've solved this one already. And let me tell you guys why I do believe that this ARK AGI benchmark, which is supposed to be the benchmark that is resistant to memorization, and of course the benchmark that proves whether or not we're on the right track to AGI, has actually been solved. So do you remember the video I posted yesterday? Yesterday I spoke about the surprising effectiveness of test time training for abstract reasoning. And the main takeaway from this paper was the fact that they actually get state of the art on public validation accuracy, which matches the human score. Basically, on the ARK AGI benchmark, or for the public set, they managed to get state of the art, which matches human performance. And this was people from mit. So MIT who tested on this very tough benchmark, they managed to get 62%, which was a variation of test time compute, the same method that OpenAI are using. So think about it like this. For Cholle's evaluation, which is the toughest evaluation there is for LLMs, people at MIT managed to get 61.9%, which matches the human score. So we have to think, what do you think OpenAI have done with that benchmark, considering the fact that they've managed to create an entire model around this method? And it's quite likely that Future iterations of O2 and O3 are going to even surpass what we think is possible. Now, what's crazy about this super difficult benchmark is that currently, for this benchmark, you guys really need to pay attention here is that for this benchmark, okay, the highest score that we currently have is an approach that is pretty different. Okay, so take a look. Okay, so you can see that the first one is ryan Greenblatt. And two steps down is 01 preview. Now, the most interesting thing about this is Ryan's approach. Okay, Remember how I said in the beginning of this video, one of the main critiques to OpenAI's GPT series model was the fact that that model doesn't learn. And it, according to Gary Marcus, needs a neuro symbolic approach, which is something that it doesn't currently have. And basically Ryan's actual approach, which is a method of tackling the Arc AGI benchmark, he actually uses GPT4 and he actually uses a neuro symbolic approach. And you can see he managed to get the highest level benchmark. Now, his approach, which actually gets the highest level benchmark is using an LLM with a discrete program search, which is actually very similar to what OpenAI have done with 01. So basically extrapolating this information all out on one side, we have Gary Marcus basically saying that, look, deep learning is hitting a wall. I told you guys, you guys need neuro symbolic AI. I'm completely right about GPT slowing down. On one side. He is right, because the GPT series is slowing down. But what it seems to be happening now is that there is another paradigm that is quickly gaining traction. Because if we take a look at this current evaluation, one that current LLMs really struggle with, we've already seen people at MIT manage to crush this benchmark and achieve basically human level reasoning. And we've already seen someone use a neuro symbolic approach to get 72% on one of them and 43% on the private one, which is essentially what OpenAI are pursuing. So if that is the case, I wouldn't be surprised if OpenAI and Sam Altman's team have already cracked this incredibly difficult, difficult evaluation, because we've seen that other individuals have really gotten close to human level performance. So what's to say a lab with billions of dollars wouldn't have managed to find even greater techniques than what is out there publicly? And what's crazy about this is that this same technique that everybody's now talking about is one that is known for being used in AlphaGo, which is of course one of the first superhuman techniques that was able to get ridiculously level superhuman performance in a game. And I'm pretty sure you're all familiar with how crazy AlphaGo is. So basically, I think this is a situation where I guess everybody wins. Because while some people are arguing whether AI is slowing down or it's speeding up, I think what's quite likely is that we have another S curve paradigm. The GPT series paradigm is potentially slowing down. While some people state that this isn't the case, I think evidence from all three Frontier Labs is pretty hard to ignore. Even if they are leaks, it's not like there is any company that's going to come out and publicly admit that that their models are slowing down in performance, even if it is a prior series that they might not be focusing on. It's quite likely that the O1 or this test time compute paradigm is going to be the future of AI that reasons in an advanced way. And I would honestly think that this is something that the majority of normal people don't need to worry about because 010203 those kinds of models are going to be used in areas where you only really need super advanced reasoning, which is not something that you use on a day to day basis. Most people are going to be pretty happy with a GPT4 level model that's able to perform well on your day to day tasks. And when it comes to benchmark saturation, I wouldn't be surprised if benchmarks are saturated. Because when we actually look at what Sam Altman's basically saying, if we remember the test time graph, we remember that these models get better when submitting more samples. For example, when O1 was allowed to submit 10,000 samples per problem, it achieved a gold medal threshold on one of their competitions. So let's think about this for other problems. What would happen if the model is able to think or submit 100,000 samples for a problem that's even more difficult? Could it increase the benchmark by let's say 2% each time? It's quite likely that those benchmarks will become increasingly saturated as we increase the submission size. Overall I think this statement there is no rule is pretty accurate. And I do think that currently the state of AI is very confusing because on one side, yes, some people are right that AI is slowing down, but on the other side they are right that this next paradigm.