GPT-5 Is Slowing DOWN! (OpenAI Orion News)

So OpenAI may have faced another hurdle on their roadmap to AGI as apparently they are now shifting their strategy is the rate of GPT AI improvement is slowing. Recently there was an article from the Information that basically discusses what many people in the AI industry have been discussing for quite some time, and that is whether or not these systems and these models are truly effective at learning and if the scaling laws and the various paradigms surrounding AI will will continue to keep going up in intelligence or these models start to peter out in terms of what they're capable of doing. So one of the things that we could actually see here from this article that I find to be interesting because there are just a variety of different things that are discussed in this article that need to be discussed at length to understand exactly where AI is going, and there are a few things that a lot of people are getting wrong. So I want to clear some things up. But it starts by stating that some researchers at the company believe that Orion isn't reliably better than its predecessor predecessor at handling certain tasks, according to employees. So Orion performs better at language tasks, but may not outperform previous models at tasks such as coding, according to an OpenAI employee. And this could be a problem as Orion may be More expensive for OpenAI to run in its data centers compared to other models it has recently released. So essentially what they're stating here is that the next model in the series, which according to Many won't be GPT5 but will actually be called Orion, is scheduled to be apparently unfortunately not that much better than the previous model. Now I think what they're stating here is that in some areas the model does exceed, but in certain areas it just isn't reliably better. So one area that they talk about here is the fact that coding is an area where the model isn't that much better. And I'm going to dive more into this as we get on into the article, but I just want to read two more points so then I can explain to you guys what's going on. So when they state that the model isn't currently better at the other model at coding, I think this is kind of interesting considering the fact that there was also information released not too long ago coming from OpenAI about how they have an actual internal model that apparently is executing code for OpenAI themselves. So from another article, which is additionally from the information they actually state themselves that it says, for instance, OpenAI has worked on a product to handle software engineering tasks that actually might take a human hours or days and to write and execute the code for complex applications based on customers instructions. But it isn't clear whether OpenAI would launch such a product. So basically from this article, in a video that I covered around seven days ago, this was basically talking about how OpenAI have their own internal product that handles software engineering tasks. So yeah, I'm wondering what kind of model are OpenAI using that isn't the same with as to what they're using for Orion? Because apparently that one internally works really well and apparently it's actually really, really popular internally. So it will be interesting to see if this product ever makes it out or whatever kind of abilities that they've got in these really good coding models seem to manage to make it to Orion. Now I don't think, you know, Orion encoding, all that stuff is the main information. Let's get on to that. So basically, one of the main problems with this article that a lot of people are having is the fact that this article is talking about how the Orion situation could test a core assumption of the AI field known as the scaling law models. And that's the fact that LLMs would continue to improve at the same pace as long as they had more data to learn from additional computing power to facilitate that process. So basically, if you've been in the AI community for a while, you'll know that the entire paradigm around GPT models, we've had this paradigm of the fact that, look, these models are going to get consistently, reliably better as long as we have more data and as long as we manage to train them for longer and essentially make them bigger and larger. Okay? And essentially because the Orion model isn't that much better than the model that they are seeing now in terms of GPT4O, people are starting to think, okay, if this is the case, then it might mean that these AI models are going to be slowing down in terms of what they're going to be doing. So this is something that of course many people are concerned about and many people are questioning because the entire paradigm is how companies are investing. This is why they're buying more tech chips. So this is going to be something that is really important for them to know about because of course this is integral to how AI really works. Now, I'm not going to lie, I do actually agree with this article. Contrary to what a lot of people are saying, I do agree that the GPT scaling laws are potentially slowing down, but this doesn't mean that AI is slowing down at all. What I mean when I say that is I mean that like, if the GPT scaling laws in terms of like just using ChatGPT, like GPT4 and just training bigger and bigger models was the entire way to improve the models, I think that would be OpenAI's main focus. But we can clearly see now that OpenAI's main focus is the new scaling laws, okay, on test time compute, which is of course completely different. And that scaling law is completely different to the previous one of where you're just focusing on adding more data. So you can see right here it says the industry appears to be shifting its effort to improving models after their initial training, potentially yielding a different type of scaling. And that is of course something that does make sense considering the fact that right now everybody knows that the paradigm that we're looking at is of course the test time compute paradigm. So this is the paradigm that, you know, everyone knows about. It's about how AI thinks about its responses and this is how the model actually gets better. So I do think that yeah, maybe the old way of doing things like just adding more data and then just doing some fine tuning and some post training, those kinds of things probably aren't going to produce massive gains. But literally you can see here on this new paradigm, the O1 paradigm, which Sama has, you know, even talked about the O2 series, stating that, you know, that one is going to get 105% on the GPQA, I don't think that series is slowing down at all. And I think that matters more than Orion anyways now, crazily, Sam Altman said that he did expected Orion to be significantly better than the last flagship model. I still do think that Orion is going to be significantly better than the prior model. Even if it doesn't look like it, I still do think that this model is going to be significantly better than. Because every single time that a new model is made, we often do find new solutions to existing problems in terms of the efficiency, in terms of hallucinations. So it does seem that this model, even if it's not going to be remarkably better, we know that marginal improvements actually do allow for a lot more use cases. I want you guys to think about the fact that when Claude 3.5 Sonnet actually managed to produce really effective code, how many people started to produce software? How many people started to do different things with regards to Claude 3.5 sonnet? So I do think that even if this model isn't ridiculously better, because of course the jump from GPT 3.5 to GPT 4 was pretty incredible, but of course, even if that jump isn't as substantial, we know that it does unlock a more wider variety of different use cases that are going to be really amazing for many individuals in the AI industry. Now, a crazy thing as well that I didn't really see most people talk about, which was kind of fascinating, was the fact that they spoke about how though OpenAI had only completed 20% of the training process for Orion, it was already on par with GPT4 in terms of intelligence and abilities to fulfill tasks and answer questions. This is what Sam Altman said. So the fact that the model has only completed 20% of its training process and it's already on par with GPT4, I think that is a substantial statement considering the fact that this means this model is going to, you know, just based on that, I'm not trying to do the mathematics here, but if it's only completed 20% of its training process, it's quite likely that this model is probably still going to be significantly better than GPT4 in a variety of different ways that, like I said before, already is going to unlock many different use cases, just making the model even better. Now, of course, you can see here they actually talk about how Orion's performance ended up exceeding that of prior models. The increase in quality was far smaller compared to the jump between GPT3 and GPT4 according to the same employees who have used or tested Orion. So it's going to be pretty interesting to see what happens if Orion is delayed, if there are certain improvements. And it's kind of interesting because I do remember that recently there was an article that spoke about how, you know, Google were expecting huge results from their Gemini 2 model, but haven't been able to get the kind of results that they were hoping for in terms of an increased level of abilities and all these kinds of different things that we're hoping to see. So this is where I was actually talking about how the fact that OpenAI are busy baking more code writing capabilities into their model. Remember how I just spoke about the fact that, you know, they already have an internal tool that they use for software engineering tasks that that's pretty popular at the company and I'm wondering if that's going to somehow make its way into our products. And of course OpenAI are also developing software that can take over a person's computer involving web browser activity applications, yada yada yada. So basically stating, Look, OpenAI is still developing their own software engineers in terms of tools and then of course in terms of AI agents, they're still developing software that can take over a computer completely. Now, interestingly enough, there's a lot of contradictory information here because Altman and CEOs of other AI developers say that they haven't hit the traditional scaling law limits yet. So of course, if these scaling law limits aren't hit yet, that would mean that they're going to still be developing these massive data centers. And I think that is of course important because when we look at what people say versus what they do, you always have to pay attention to what people are doing versus what they say because that is going to give you a greater indication of what is truth versus false. Because of course, if the scaling laws were really slowing down, then I think we would definitely see perhaps not this major build out that we are seeing in terms of AI infrastructure. So them stating that, look, we haven't hit the traditional scaling law limits yet, we're still going to be buying these data centers. I think it's still important. But I do think that a lot of the future infrastructure is just going to be put to inference time compute because we've seen that the gains in terms of those scaling laws are not slowing down in terms of every time we add more compute to the model, it seems to get even more smarter. So it seems like maybe they are adding even more data centers because of course they know that there's going to be an increased demand and of course because they know that inference time compute is potentially going to become a more important paradigm. Now, interestingly, someone who actually works at OpenAI on the reasoning model, which is the O1 series, actually said something interesting. And I want to show you guys two screenshots. Okay? So the first thing that they said was, they said, after all, are we really going to train models that cost hundreds of billions of dollars or trillions of dollars? Which is what Brown said. Okay? And this is Noam Brown, the guy who worked on reasoning at 01, which is of course the model that thinks before it responds. And he said at some point the scaling paradigm breaks down. So basically he's stating that, look, even if these models got a lot better with data and stuff like that, it's going to be pretty hard to spend trillions of dollars to train a model because it just isn't feasible. Now the reason I said I need to show you guys another screenshot is because this actual screenshot of this article, I don't want to say it was taken out of context, but he actually did respond to the statement here. So he actually said that in the TED Talk that he gave, which they selectively quoted in the article. I make the case that there won't be a slowdown in AI progress anytime soon. So I think it's interesting to see that, you know, the actual person who said the statement is basically stating that, look, this entire AI progress isn't slowing down anytime soon. I think we can take that at face value more than what is being said in the article. And I do believe that, like I said before, even if these models, you know, slow down, and I'm talking about the GPT series, not the O1 series, and it doesn't mean that AI progress is going to slow down overall because there's still different ways that you can prompt these models and still different ways that you can interact with these models that you're going to get out more information. You have to remember OpenAI isn't the only company that is working on this stuff. You've got Google, you've got Anthropic, you've got X, AI, you've got Meta as well. So that's four other major companies that are all playing. So it means that even if, let's say OpenAI hits some kind of roadblock, it's quite likely that other companies are going to managed to still be able to make progress. And remember, this doesn't mean that AI is slowing down, it just means that the scaling laws are now focusing on a different aspect of AI. Now for those of you wondering about the naming of the model, it looks like Orion is probably going to replace the naming of GPT5. So it says when OpenAI releases Orion by early next year, it may diverge from its traditional GPT naming convention for flagship models, further underscoring the changing nature of LLM improvements. Employees said. Now of course they do talk about a data war. And it says one reason for the GPT slowdown is a dwindling supply of high quality text and other data that LLMs can process during pre training to make sense of the wall. So now this is where it actually talks about the data wall, which is essentially where in order to make this model smarter, of course one of the things that we do is we do add a lot more data. And one of the things that we do do is we do add a lot more high quality data. Now one of the problems that they are facing is of course the fact that high quality data available online on the Internet is quite hard to come by considering the fact that a lot of Internet data is just simply garbage. Like a lot of the data that you see just is Downright nonsense. Which means that of course, in order to get higher quality data, you either have to have humans make that quality data like smart humans, or you have to have synthetic data which is curated by these models. Now both of those things are pretty hard and this is potentially one of the reasons that we do have a GPT era kind of slowdown. And I think one of the things that we do have to look at when we're looking at this kind of architecture is the fact that humans don't need millions and millions of pieces of text to be able to generate high quality text. So it will be interesting to see in future paradigms how models manage to get super smart via a more efficient architecture that doesn't require these data hungry models. Now essentially, of course, remember how I spoke about one way of getting higher quality data is by introducing synthetic data. One of the things that they actually did with Orion, which was quite fascinating, was that they said was Orion was trained in part on AI generated data, but produced by OpenAI's other models, including GPT4 and recently released reasoning models. According to an OpenAI employee, however, such synthetic data, as it is known, is leading to a new problem in which Orion may end up resembling those order models in certain aspects, the employee set. So essentially what you have here is something that could be referring to model collapse, which is where when models produce data, it's based on their training data and based on their entire worldview of how the world is. And that's just course based on the training data and how they were trained with human reinforcement, human feedback. And basically what happens is that if you have a model produce data to another model, it's quite likely that that model that you're now training on, that data is going to resemble parts of that older model in certain aspects. Which brings us back to the problem, which is why synthetic data generation is really hard, because you have this, I guess you could say, kind of family tree of models where the first model is really diverse and then because you use some of that data to train the other model, it just becomes less and less diverse over time because it's training on the same. So I think one of the things that you know, of course you have to do is you have to get really diverse pieces of data. And interestingly enough, the article also quotes where they talk about the fact that there's a GPT slowdown, slash, asymptote.

There is this very interesting asymptotic kind of thing that's happening right now where you know, two years ago there was one LLM that was like way out ahead of everybody else's, which was OpenAI's. And sitting here today, there's like six that are like on par with that. And interestingly, at least for right now, they're all set of asymptoting at this, at sort of the same point. They're kind of hitting the same, the same ceiling on capabilities. Now. They, you know, there's lots of smart people in the industry working to break through those ceilings. But you know, sitting here today, if you just looked at the data, if you just look at the charts of performance over time, what you would say is there's, there's at least a local topping out of capabilities that's happening, right? Yeah. And if you look at like the improvement from GPT 2.0 to GPT 3 to 3.5 and then compare that from like 3.5 to 4, you know, we've really slowed down in terms of the amount of improvement. And the thing to note on that is the GPU increase was comparable. So we're increasing GPUs at the same like rate, but we're not getting the intelligence improvements at all.

Now one of the fascinating things that I actually saw here, and I think this is going to actually be a problem for OpenAI and I don't know how they're going to solve this, but it will be interesting to see how they do. But this is basically the fact that Everyday Joe doesn't need AGI and it says, mathematicians and other scientists have said that O1 has been beneficial to their work by acting as a companion that can provide feedback or ideas. But the model is currently priced six times higher than non reasoning models and as a result it doesn't have a broad base of customers said to employees with knowledge of the situation. So the model is advancing scientific research, but apparently it's too expensive for people to use at this moment in time. I do think that this kind of makes sense because of course these models actually do burn through a lot more tokens than traditional models with ChatGPT, GPT4, GPT, you know, mini, whatever. When you interact with these models, essentially what happens is you interact with the model, it produces, you know, a thousand tokens or whatever. But with these other models that are thinking there are actually so many tokens that are going on for the model to think about its response and essentially with that it produces a lot more tokens which of course aren't free, they cost a lot and so therefore it means these models are six times higher, which does mean that right now it isn't cost effective. I do think, think however that in the future it's quite likely that these models are going to become a lot more cost effective in terms of their pricing. And I do wonder how that translates to economic value for individuals around the world. Will people start using this everywhere and everywhere? I think that that is quite likely the case considering the fact that it is definitely beneficial to scientists. So it's going to be interesting to see that, okay, we've got the O1 series reasoning of models that are really smart and advanced that for like things like physics, chemistry, maths. And then of course you've got the GPT series models that are for things like maybe even coding, content creation, literature, storytelling, those kinds. Interestingly enough, some people have pointed out that Gary Marcus has claimed victory. And if you haven't been paying attention to Gary Marcus, he is what some would call an AI skeptic. Now I don't think he's an AI skeptic per se, considering the fact that he's, you know, built his own AI company and then sold it to the likes of Uber. But he said, folks, over game I1, GPT is hitting a period of diminishing returns, just like I said it would. And I think this one was particularly interesting because Gary Marcos was basically saying that look, at the end of 2024 we're going to have a bunch of GPT4 style models and there's going to be no major improvements. So it will be interesting to see exactly what happens with the next series of models. Orion, maybe the next version of Claude, maybe Gemini 2. It will be fascinating to see what happens to that regard. Now also, Peter Linda, the VP of products, basically said that people underestimate how powerful test time compute is, compute for longer in parallel or fork or fork and branch arbitrarily, like cloning your mind a thousand times and picking the best thoughts. So I think of course this is going to be something that becomes more powerful over time because of course if a model is able to generate 10,000 thoughts and pick the best one, that's going to become a lot better than just picking the first initial thought. And over time I do think that paradigm will exist and these models will continue to get smarter. So overall, basically, to summarize this video, some people think that the paradigm is slowing down. But I think that even if the GPT series paradigm is slowing down, I do think that the O1 series is about to go crazy. So that means that even if you only get marginal improvements from here on out, there's still going to be ways to prompt the GPT series, and there's still going to be improvements discovered over time to make these models more effective. So the narrative that AI is slowing down completely isn't really true, considering what we know now.