The AI Hype is OVER! Have LLMs Peaked?

So one of the current questions that I've been seeing floating around on Twitter and on social media is that AI hype might be wearing off. And when I'm talking about AI hype, this is referring to the trend that generative AI has, I guess you could say, reached an exhaustion point. Now, I disagree and I'm going to tell you guys why, but I'm going to go over some of the key examples and what people are stating. So essentially there was a clip here and a lot of people were stating that this clip from Sam Altman, people are stating that the hype is wearing off, the vibes are shifting. You can feel it. And basically in this clip here, Sam Altman, literally, he's only really stating that, you know, I don't care if we burn $50 billion a year, we're building AGI and it's going to be worth it. So it's not a crazy, crazy statement. I think why people are stating that the hype is wearing off and that vibes are wearing off and this video is going to be. And essentially what this is referring to here is the fact that, you know, I guess a lot of people are thinking that, you know, I guess you could say that generative AI is currently slowing down in terms of the capabilities and what we're likely to see in the future. Now, like I said before, it's just mainly due to this clip and a bunch of other different factors. But let's get into how cycles work in terms of technology and a graph that a lot of people have been referencing when they talk about LLMs plateauing. So something that I also saw quite a bit being, you know, passed around as, like, this infographic is of course, the Gartner Hype cycle. So essentially, essentially it's just a graphical representation used to illustrate maturity, adoption and social application of specific technologies. It basically just provides a conceptual framework that helps stakeholders and individuals understand how technologies evolve from the time they are introduced until they reach mainstream adoption. And the hype cycle can be particularly useful for evaluating emerging technologies like generative AI and of course, large language models. So, of course, we essentially have the technology trigger. This phase initially occurs when a new technology is conceptualized or when a significant development makes it publicly known, creating interest and significant media attention. For example, this is when GPT 3.5 ChatGPT was released and it demonstrated the ability to create text in the long format. And then of course, that's when we got to GPT4, which is stage two. So this is where there is the peak of inflated Expectations. Now, I've got to be honest, it's not just LLMs that were going crazy at this point. There was also 11 labs and other image generation services like Midjourney and of course other services like Stable Diffusion. So I would argue that the problem with this is that with generative AI experiencing some kind of increase in the actual media coverage, I would say that this is something that is cumulatively increasing in terms of the expectations. And that's because like I said already there were many several different categories that came together at the same time. And of course this could include LLMs being at the peak of inflated expectations. Now, I want to say that I do disagree with this hype cycle for AI. I do think that this is nowhere near it, its peak where it should be. But there of course is inflated expectations. When a new technology comes to fruition, a lot of people may exaggerate what the technology can really do. For example, some people say it's going to replace entire careers, automate entire tasks and revolutionize entire industries. Now, whilst, yes, that might happen in the future, I don't think that that is completely happening with GPT4 and GPT 3.5. So some are arguing that this is where we are at currently. Okay? And then of course the trial of disillusionment. And this is where technologies enter this phase when they fail to meet the inflated expectations and stakeholders become increasingly delusion. So issues related to the technology start to arise, such as the biases in large language models, the need for vast data training, the high operational costs, the high inference costs, the environmental impacts, they become more apparent and then they become criticized. And one of the main things that many people are talking about, about with LLMs is of course things like the hallucination and the high inference costs and of course the training costs, because these models are certainly not cheap. And it seems like there's only a few companies that can run and train these large models. Now of course there we have the slope of enlightenment. And this is essentially where with LLMs, all of these previous issues get ironed out. So things like the issues of hallucination and the biases, they get ironed out right here. And this is where as more experimentations, as more implementations occur, the market matures and second third generation products appear. And with LLMs this would involve developments that address the earlier criticisms, such as improving the model efficiency, the inference costs, reducing the biases, and of course the reliability of the model. And this is arguably where people think AI is going to go. The technology becomes stable and accepted and this means widespread adoption. However, I think the graph isn't going to look anything like that. I, I think it's probably going to look something like this where we go up, we dip a little bit and then we continue a trajectory upwards. Because like I said before, whilst, yes, that it does seem to many different pieces of statistics that it looks like we are slowing down in terms of AI versus human performance. And even on the Stanford AI Index, we can see that AI has surpassed human performance on several benchmarks, including some on image classification, visual reasoning and English understanding. Yet it trails behind on more complex tasks like competition level mathematics, visual common sense reasoning and planning. And you can see here that if we take a look at actually how the AI is moving, we can see that in image classification, visual common sense reasoning, natural language inference, all of these seem to be coming towards the human level baseline, but they don't seem to be going upwards like in a crazy level on the graph. And some people would argue that this is because AI, generative AI, large language models, whatever you want to call it, if you just want to group everything together, that we have reached our limit in terms of where we are and new architectures are going to be needed. Now, I would firmly disagree with this for several reasons that I'm about to explain and I'm going to show you guys some really key evidence on why things are about to actually get very, very crazy in the world of AI and why we're about to go into a very, very abundant era due to AI. So one of the things that many people are not actually taking into account is the fact that currently there are many different things going on in the world of AI that people aren't paying attention to. Some people are just paying attention to large language models. And this doesn't make sense because there are vast and many different categories in which AI is currently exceeding, for example in voice recognition and in voice generation. OpenAI has recently developed their voice engine, which was actually something that they developed in around 2022 and it was basically a state of the art system that could recreate anyone's voice. In addition to that, if you do remember, OpenAI also did talk about and quote unquote release their SORA model to showcase us how far they've come in creating a text to video model. Now I think you have to understand how crazy this is because for people to say that generative AI is stagnating is a crazy statement, when literally this year we literally got sora, which blew everyone's minds in terms of the capabilities. So with Sora, I still find it absolutely incredible that people can say AI is stagnating. Because with Sora, as you've all seen, this was something that was truly just mind blowing. This piece of technology showed us how crazy it is when you get a group of AI researchers dedicated to doing something. And I'm going to give you guys a quick memory jog. Remember that OpenAI isn't a company that's focused on video generation. This is just a subsection of their company. So the video generation aspect was something that I guess they just wanted to see if they could do well at. And they literally surpassed state of the art models, Google Runway Picolabs, they completely surpassed them. And this is absolutely crazy. I mean the demo literally, absolutely shocked everyone. I was truly, truly speechless when this technology was here. And I'm someone that pays attention to all of the AI news. And of course we do have Devon. Cognition Labs released Devon, their first AI software engineer. And this was a AI wrapper around GPT4. But it did a few things in a unique way where it was able to surpass GPT4 on certain software engineering benchmarks. So one of the first things the point here I'm making is that you might think that LLMs are stagnating, but that is not the truth at all. We're going to get into LLMs later, but overall, Voice Engine, Sora and Devin show us that generative AI is really, really not going to be stagnating anytime soon. But if that didn't convince you, let me show you guys some of the recent statements that show you that we're absolutely in for a crazy ride. So Sam Altman recently said in an interview at Stanford's Entrepreneurship Talk, he spoke about GPT4. Now remember, GPT4 currently is a state of the art system, meaning that it is the best of the best that we can currently get our hands on for public use. Which means that Sam Altman currently probably has access to frontier models that are being developed by OpenAI. And remember, the big labs like Anthropic and Google are currently behind them in terms of what they're creating. So take a listen to this statement.

OpenAI is phenomenal. ChatGPT is phenomenal. Everything else, all the other models are phenomenal. It burnt. You've earned $520 million of cash last year. That doesn't concern you in terms of.

Thinking about the economic model of how.

Do you actually where's going to be the monetization source?

Well, first of all, that's nice of you to say, but ChatGPT is not phenomenal like ChatGPT is mildly embarrassing at best. GPT4 is the dumbest model any of you will ever have to use again by a lot. But you know, it's like important to ship early and often.

So if you weren't paying attention there, Sam Altman literally just said that this is going to be the dumbest model that we will have to use or that we will have had to use by far. So he didn't just state that this was going to be an incremental increase. He clearly stated that GPT4 was dumb. He stated this model was not, you know, great. He stated that this model was not that good. He clearly stated that GPT4, a current state of the art system that people were literally able to get, you know, increasingly capabilities just by wrapping this system and being able to do software engineering tasks. And a lot of people even using GPT4 to be able to train robotic systems. Like recently we saw in a research paper and this was literally where we had language model guided SIM to real transfer. So we basically had large language models where basically writing the reward functions for an AI system and it was able to do it very effectively. And it literally just went from simulation to real life, which means that this is going to immediately speed up how quickly we're able to train robots and have them, you know, doing well in novel environments. It's pretty, pretty insane. So it's an LLM guided SIM to real approach. So that is pretty crazy. And remember, I'm guessing that they were using GPT4. That's actually a study that I haven't covered just yet. But the point is, is that Sam Altman states that the current state of the art model, the current model that other industry labs are trying to be with millions and millions of dollars, he states that that model is bad and it's dumb. He didn't just say it was kind of smart, he said that it was dumb. Which leads me to believe that we are truly, truly not even scratching the surface for what AI systems are. He could have just said future models will be interesting. He could have just said they will be, you know, kind of good. But he literally said, okay, I'm gonna play it one more time.

First of all, that's nice of you to say, but ChatGPT is not phenomenal like ChatGPT is mildly embarrassing at best. GPT4 is the dumbest model any of you will ever, ever have to use again by a lot. But you know, it's like important to ship early and often.

So that just goes to show how clear there's going to be a distinction in the future. Now, he also stated that there will be a massive jump from GPT 3.5 to 4, and there will be a similar jump from GPT 4 to GPT 5. So it's important to know that the jump from GPT 3.5-GPT 4 was incredible. Because of GPT 3.5's limitations, it meant that it couldn't be extended to certain tasks. But if we have that same jump from GPT4 to GPT5, then things are truly about to change.

There's so many questions. First of all, also amazing. Looking back, it'll probably be this kind of historic, pivotal moment with 3, 5 and 4.

With ChatGPT, maybe 5 will be the pivotal moment. I don't know.

Hard to say that. Looking forwards, we never know. That's the annoying thing about the future. It's hard to predict. But for me, looking back, GPT4, ChatGPT is pretty damn impressive, like, historically impressive. So allow me to ask, what's been the most impressive capabilities of GPT4 to you and GPT4 Turbo?

I think it kind of sucks.

Typical human. Also gotten used to an awesome thing?

No, I think it is an amazing thing. But relative to where we need to get to and where I believe we will get to, at the time of GPT3, people were like, oh, this is amazing. This is. This marvel of technology. And it is. It was. Now we have GPT4 and look at GPT3 and you're like, that's unimaginably horrible. I expect that the delta between 5 and 4 will be the same as between 4 and 3. And I think it is our job to live a few years in the future and remember that the tools we have now are going to kind of suck looking backwards at them.

So you can clearly see here that he's basically stating that a year from now, when we have GPT5 or the next level of Frontier model is released, we're going to look back at GPT4 and think it's pretty, pretty bad. So as much as people are stating that AI hype is wearing off, Sam Altman's feeling it. There's this Gartner Hype cycle. It's important to remember the subtle cues, the subtle tells, the subtle statements that we've clearly seen from industry leaders about these future, future models that only they currently have access to. Now. If generative AI was actually slowing down, it would be due to a bottleneck and there are only a couple bottlenecks that are really the issues. Okay, One of the issues is of course energy, and this one isn't really talked about enough because energy isn't something that people think of. But trust me when I say these inference costs are really, really expensive. And a lot of people, you know, including Mark Zuckerberg, are basically stating that one of the major limitations for future AI systems and where things might actually start to slow down is due to the energy cost, the rampant energy cost for these AI systems and just how much electricity they consume. And there were even a few rumors Talking about a GPT6 training cluster project that would arguably shut down the power grid, something along those lines. I know it does sound crazy and I know rumors are rumors, but energy is expensive and a lot of it is required to run inference on these large language models, which is why they often restrict us.

I think there's this issue of GPU production. Yeah, right. So even companies that had the money to pay for the GPUs couldn't necessarily get as many as they wanted because there was, there were all these supply constraints. Now I think that's sort of getting less. So now I think you're seeing a bunch of companies think about, wow, we should just like really invest a lot of money in building out these things. And I think that that will go for, for some period of time. There is a capital question of like, okay, at what point does it stop being worth it to put the capital in? But I actually think before we hit that you're going to run into energy constraints.

Right?

Because I just, I mean, I don't think anyone's built a gigawatt single training cluster yet. I mean, just to I guess put this in perspective, I think a gigawatt, it's like around the size of like a meaningful nuclear power plant, only going towards training a model. And then you run into these things that just end up being slower in the world. Like getting energy permitted is like a very heavily regulated government function. And if you're talking about building large new power plants or large build outs and then building transmission lines that cross other private or public land, that is just a heavily regulated thing. So you're talking about many years of lead time. So if we wanted to stand up just some like massive facility to power that, I think that's a very long term project.

So basically what Mark Zuckerberg is stating here is that energy is going to be a huge bottleneck because unlike software where you can make it more efficient and you can do things quickly or you can get a GPU produced quickly trying to build a nuclear power plant. It takes time trying to build these kind of infrastructure. It's not something you can just do in a day. And it's something that's quite important when it comes to generative AI, because as the race continues, they're going to be spending a lot more money. They're going to be spending a lot more in terms of how much they're spending on cooling, because these GPUs heat up quite a bit. So I think this is something that, you know, this is probably where the constraints actually will come from in the future, and this is where things might actually slow down a little bit, because there are going to be a few things that make this truly harder to continue with.

I think we would probably build out bigger clusters than we currently can if we could get the energy to do it. So I think that that's.

That's fundamentally money bottlenecked in the limit.

Like, if you had a trillion dollars, it's time. Yeah, Right. But it depends on how far the exponential curves go.

Right.

Like, I think a number of companies are working on, you know, right now, I think I like a lot of data centers on the order of 50 megawatts or 100 megawatts, or like a big one might be 150 megawatts. Okay. So you take a whole data center and you fill it up with just all the stuff that you need to do for training, and you build the biggest cluster you can. I think you're. That's kind of. I think a bunch of companies are running at stuff like that.

So it will be interesting to see where generative AI does fall down. But like I've said, it's not slowing down anytime soon. And if you remember, OpenAI and Microsoft are building a $100 billion Stargate AI supercomputer to power the AGI or ASI. Now, there's also another bottleneck, which is what I've titled here, which is the Compute Problem. Essentially, this just means that the compute capacity for AI systems is far too great and it kind of exceeds the demand that we require. So that's why they're building out this $100 billion supercomputer to meet the demands of future generative AI systems or to essentially power the next industrial revolution. Because, my, oh, my, if an AGI or ASI is here, it's going to be used pretty much everywhere to power the economy, and you're going to need the compute and the infrastructure to do that. And Currently, this is one of our biggest things, because OpenAI and Microsoft don't even have enough chips and they don't even have enough compute currently to compete with the likes of Google. So what we have here is we have updates to the chips and you can see that Nvidia's recent Blackwell is pretty, pretty incredible. And this was one of the most important developments for AI because this accelerates the training of large language models and generative AI systems. So the Blackwell GPU architecture, with its 208bit billion transistors and its enhanced transformer engine, is designed to dramatically increase the training of large language models like GPT4. According to Nvidia, Blackwell can provide up to 30 times higher performance for generative AI inference compared to the previous H100 GPUs, 4 times faster training performance for large language models. And this essentially means large language models like GPT4, which took around 90 days to train on 8,000 H100 GPUs, consuming 15 megawatts of power, could potentially be just trained in just 30 days on 2,000 black core GPUs, only using 4 megawatts, which is pretty, pretty incredible. And that curve definitely reminds me of some that we've seen before where things are starting to exponentially increase. So the point right here is that these are the actual bottlenecks of generative AI, because a lot of people are thinking that, you know, this, things are slowing down, things are getting worse. And trust me, guys, if you've been paying attention, things I would argue are actually speeding up internally. And one of the things that I didn't even include in this presentation because I forgot about it, is the fact that right now it's like OpenAI lit a match under every other company, because now other companies are realizing that, whoa, there's a huge, huge AI race going on. And if we partake, we could definitely be getting billions and billions of dollars. And that means that other companies and other startups are all rushing down the corridor to see if they can get a piece of the pie, which means that we're about to see a complete revolution and a complete new industry in terms of all of these products and services. Now, one of the biggest things that I think that most people need to consider is that OpenAI are no longer open. Okay? And if there's one thing you take away from this video, please understand this, okay? Things might be slowing down externally, but things are not slowing down internally. And what I mean by that statement is that currently we're at a stage where things have moved from an open research environment to a Closed research environment. The reason this has happened is because OpenAI, they're no longer essentially a company that's just focused on research. They are a business. And businesses, you know, they hide their secrets and they hide their innovations because they don't want their competitors to have them. If OpenAI shared all their secrets, then other companies could easily build GPT4 with remarkable accuracy. And OpenAI essentially has secret sauce. The thing is, OpenAI also doesn't publicize their research. I'm sure breakthroughs are made every single month. Okay? And you have to think about it. If OpenAI did their, whatever they did with GPT4 so long ago, they must have some secret kind of breakthrough, they must have some secret source and they must have something that others don't, which essentially means that OpenAI are making consistent breakthroughs. And remember Sora, they had Sora, we had no news, no indication that they were even developing some video AI. There was literally no indication whatsoever. There was no interview from Sam Altman. There was literally nothing we could have picked up on, on the fact that they were even training such a model. And boom, they just, you know, put it out into the open. The point is, is that we know, we have no idea on what's going on at, you know, closed AI, OpenAI, whatever you want to call it. The point is, is that internally I can guarantee you guys they are like two to three years ahead from where they are. And, and the point there is that whilst you might think, ah, they haven't released anything in a while, that doesn't mean things are slowing down. It just means that they're thinking, okay, how can we not shock the public with this next release that we know is literally going to take everything. Remember, other companies are still playing catch up. GPT4 finished training in August of 2022, which means that we are very, very lucky because we're going to be in for a real surprise when GPT5 gets here. Now another thing to note as well, okay, is that GPT4 being the benchmar, not mean plateau. The problem with this, and like I said before, this is a business, which means things are going to change. GPT4 is currently the benchmark, which means that companies are incentivized to train their models to surpass GPT4 and then release that model. The reason this creates the illusion that things are plateauing around GPT4 is because these companies are no longer incentivized to, to go duly past GPT4. They're only incentivized to just beat it. And that is because of course with GPT4, that is something that people state, oh, it's the best system. It's the best system. So if another company like Gemini or Anthropic or Google can come out and say, look, Our system surpasses GPT4 and all benchmarks, they're going to immediately release that model after it's fine tuned or after it's whatever they've done with it, and then run with that so that they can market that and get the customer base. Because they know that OpenAI are waiting to release GPT5 potentially after the elections, and that gives them some time to reclaim the market share. Understand that where GPT4 is is just an indication of where other models are going to stop. And if you think that that is just a pure speculative argument, look at how close GPT4 is to some of these models. You have to understand that if they didn't beat GPT4, they wouldn't be releasing these models. This is 86.4%. They literally got it up to 86.8%. Another one here, 92%, they got this up to 95%. Okay. It's not like it's completely surpassing them. And I guess some people, like, look, it all, you know, slows down around here. No, they just want it to be as close as possible so that they can get this out as quickly as possible. Because they know by the time Open Air releases next again, they're going to be even behind. And you can see here that Gemini Ultra. A lot of people were even debating this because this was chain of thought at the 32 and that's what they did in order to beat this. Because I'm guessing that when they had finished training the model and when they finished fine tuning it, they had to increasingly develop certain methods just to get this metric right here. And that's why I state that GPT4 being the benchmark does not mean we're currently at a plateau at all. Because it's likely that these companies are just benchmarking their models up to GPT4 so that they can get them out. Now here's why things are going to go even crazy. And remember I said this because the next 365 days are going to be absolutely insane. Agents are still early, okay? And someone actually recently created a benchmark where they're talking about multimodal agents for open ended tasks in real computer environments. And essentially with this you can see that humans can accomplish over 72% of the task. And the best current AI agent can only do 12.24%. What happens when AI agents can get to above 80%? That is truly going to change everything. And with the advanced reasoning and with the advanced capabilities of future models, we are going to see a future that we'd never seen before. Now, there was also something that I covered in a previous video that I'm guessing the majority of people are just completely glossing over. And I'm sure it's because during the video I was kind of sick because I had some kind of flu, whatever, but I still made the video anyways. So essentially there was this thing right here. Okay, this is Mesa's kpu. Now, this is a little bit speculative because they haven't released too much information, but if you check the benchmarks here, you can see that this surpasses Claude 3, Opus, Gemini Ultra and Mistral large at all benchmarks, okay? And this is because they use a advanced reasoning engine on top of GPT4 Turbo. Now, this is pretty interesting because this shows us that we are still very early on the reasoning capabilities. Which is why I argue that Sam Altman here says that GPT4 is dumb and why he also says here that it was not a very good AI system. So there was one demo released by Mises KPU in which they showcased an AI system and actually doing reasoning with an advanced task. And recently on their Twitter, I'm not sure why it's not getting any love or any actual, you know, tweets about it. They've shown that this system is able to, you know, use some reasoning steps. And this is their system. They're messing around with it and they're showing that it's able to complete a lot of tasks really, really correctly. So you have to remember that internally things are going at light speed. Things like Q Star, things like other companies now trying to get a piece of the pie and incorporating different reasoning engines on top of GPT4 are going to push things further. And remember, it was recently that andrewng actually spoke about agentic workflows and basically said that GPT 3.50 shot was 48.35. And of course, it was Andrew Ng that did some research and found out that GPT 3.50 shot was 48.1% correct. GPT4.0 shot does better at 67%. But the improvement was dwarfed. By incorporating an iterative agent workflow and wrapped in an agent loop, GPT 3.5 gets up to 95.1%. So the point here is that there's still a lot of different architectures that we haven't fully explored with some of the AI systems that we do have, which means that we are far, far, far away from any sort of plateau. And things are going to keep increasing. Number one. We've got the GPUs increasing in terms of efficiency, we've got the data centers, we've got all of these things getting increasingly better. And of course, we've got the fact that internally OpenAI, they are blisteringly so far ahead that I'm guessing that things are going to be shocking when they are finally released.