HUGE AI NEWS: AGI Benchmark BROKEN ,OpenAIs Agents Leaked , Automated AI Research And More

So there was actually some really interesting news that was only published a few hours ago. So this is going to be an interesting one because this is news that you might not have heard. But let's get into all the details for today's summary on the last few days on AI. So one of the things that I actually did cover before was of course AI towards fully automated open ended scientific discovery. So this was something that was pretty insane because what this kind of was that we're moving towards this recursive self improvement area here where we've got systems that can literally do research on AI with current models. So basically an AI does more research, the AI improves and the improved AI does better research which can improve itself even faster. And of course that's the entire cycle. So essentially I know that I made a video on this as well and there's going to be a reason why I'm actually bringing this up again because usually I don't cover stories twice, but I'm going to talk about why. So it says we're excited to introduce the AI scientists, the first comprehensive system, so fully automatic scientific discovery. And this enables foundation models such as LLMs to perform research independently. So this is pretty crazy because if you can have LLMs performing research independently, of course there's the theory that you're just going to be able to have these LLMs like one here, one here, one here, instances of these LLMs and all generating papers, paper, paper, paper, paper, paper. And then of course from that you're going to essentially have this area where rather than there be, even though right now there is just a flood of papers flooding arcsive like literally the number of papers that have been published to arcsif has just, it's literally gone up like this in a graph. Like it's actually pretty crazy. But I mean when we have actual AI scientists automating that, I mean the graph is going to literally just go straight up into the sky because the amount of research that's going to get done is going to just be like it's probably going to be a new paper every, you know, hour or every second. I mean, how crazy is that going to be? You know, once we have, you know, systems that are efficient, once they are fast, once they are scalable, I mean, it's going to be crazy. It's going to be really, really crazy. But their methodology is here. You can see that they've got, you know, this entire methodology where, you know, generate a plan, novelty check, and this is basically where they check that the idea hasn't been done before. They score the idea, they've got the experiments, all of this stuff right here. Then of course, at the end, they've got the LLM paper reviewing. So they've got this comprehensive process on how they work through from start to finish. And I think overall, what we're going to see here is probably going to see this thing being built on top on, because once again, like I said in the previous video, this is something that is open source. So if you want to experiment with this, this is going to be something that you can do. So I'm wondering if other people and other organizations are probably going to be building on top of this. Of course, they spoke about the bloopers and the future implications. So the crazy thing about this as well, and this is kind of like the juicy thing here, and of course, this one, the research, it wasn't like groundbreaking breaking news and stuff like that, because these LLMs can only conduct essentially preliminary research. But the craziest thing about this was that there was a leak. And I say leak because it's not really a leak, but there was like, you know, some screenshots about, you know, some open OpenAI subdomains. And I'm going to show you guys that now because I wanted to talk about this, but I didn't have the opportunity to, because I didn't just want to, like, you know, just put it randomly in a video. But I found the, you know, time now. And essentially what we can see here is that we have this. Okay, so you can see here, and this is Jimmy Apples that tweeted this on, you know, August 13, 2024. And this is not news. That's very, very recent. That comes later in the video. But you can see here, he says, in a way, this trolling is rather funny. It puts OpenAI in an awkward spot with all these expectations and having to balance between patience and hype. And of course, he tweets some stuff here, but essentially what we can see here is that this is some internal OpenAI links. And you can see it says. Okay, it says scientistinternal.OpenAI.org now, what we can see here is that these dates are quite old. You can see that some of these dates are from literally March, some of them are literally from January. And Jimmy Apples tweeted this again. Now, I think he tweeted this because of course, you know, maybe they're working on, you know, internal scientists. Are they working on internal assistant scientists. Are they working on, you know, like a leaderboard to see, like, how good these Scientists are, is this something that is going to be worked on, you know, continuously? I'm not entirely sure, but I do think that this shows us that, you know, this kind of thing is of course going to be worked on in the future. OpenAI is like extremely secretive and what we do know is that like with current models, we're always essentially two years behind. And the fact that like they could have a finished product right now, but we're probably going to get it at least 18 months in advance. Not 18 months in advance, but I mean 18 months in terms of a delay. Because of course you've got to do things like safety testing, you've got to do things like red teaming, you've got to do things like post training and all these other things that involve making an AI system. And not only that, OpenAI also makes these into products as well. So OpenAI is not just like, okay, we've made this huge model here. It is, they actually make this into their product. Like OpenAI is a product driven company, it's not just a research organization, although that is a large part of what they do. A lot of people do forget that this company is a product based company. That's why ChatGPT is so nice to use. The UI is really nice. But the point here is that, you know, this is going to be some stuff that does happen. Now I do want to say here that I don't know if this is correct when I said the dates before, because I do know that Americans, they do swap the dates around. But yeah, as I was saying, considering the fact, I actually did just get this wrong. Apologies. You know, I usually don't get things wrong. But we can see that these dates are actually current date. So this is actually updated recently. So what I meant to say is that it seems like OpenAI is probably working on this now. So they're probably okay. Just from the links. Of course, links don't really mean anything. You don't really want to be doing too much speculation. But a scientist internal, a health scientist internal and an assistant API scientist. I mean these could potentially be things like a health scientist. We do know that LLMs, they have a really, really effective time at diagnosing health conditions and providing that kind of support. And of course them recently we've seen, with that research, we've seen that they've been doing amazing stuff there. And if you want to know about the health scientists, Gemini has been doing some amazing stuff. They've got like 90% on a range of different benchmarks. It's pretty Crazy. So, I mean, considering OpenAI have usually been ahead of everyone and considering that these dates are fairly recent, considering that these have been updated and some of these look like future dates, I'm not saying that they're publication dates or anything like that, but consider that these are around, you know, a couple months from now, like two months from now. We can see that this one is in October 29th. So I'm not entirely sure what this is, but I mean, if we just try to deduce what this could be, I think it could be a range of new agents that OpenAI is working on. But then again, pure speculation. And I do think that this stuff is going to come anyways, if not from OpenAI, definitely from Google, because they've been working on that stuff. But of course, you know, OpenAI is a bit more secretive. But this is definitely something that I wanted to show you guys because I do think that this stuff, and I do apologize for talking about this for so long, but I do think that this stuff is definitely, you know, coming. And this is probably what OpenAI has been working on. They're honestly working on a lot of stuff.

Now.

Something that most people did miss as well is that Luma Dream Machine 1.5 is the new text video model that is going to be launching next week. Okay, so there have been some examples. Some people have been, you know, showing you what's been done. The reason, you know, people like Luma Dream Machine is because this is a model that is free for the most part. You know, it's a lot cheaper than other models. And what Luma Dream Machine actually allows you to do that most other ones don't, is that they allow you to have a start image and then an end image. So you actually do get a lot more controllability there. And Luma actually managed to roll this out before a lot of other companies rolled out there. So this is going to be something that once again, it changes the space. Because when you have models that are really cheap and really near free, in a sense, we will actually start to see an explosion of, of this kind of content around now, whether or not you think that AI is good or whether or not you think the AI is bad, I do think the AI is coming, but that is going to be something that most people are going to have to deal with as there's no point rejecting technology. It is something that is going to be here to stay. Now, some pretty big news, and this is really what I wanted to talk about. Four hours ago, there was a new ARC AGI high score of 46%. If you remember, this is actually the benchmark that you do want to track, if you actually want to track where frontier models are in terms of their reasoning. And that is because the ARC AGI benchmark is one that doesn't measure intelligence the way that traditional benchmarks do. What the Arch benchmark does with its 85% score, that is the human baseline. And what that tries to actually do is it tries to actually measure how you can reason about problems that you don't know. And I know that might sound a little bit foolish, but I'm going to explain a little bit more. But essentially this is an article from Big Think that they did with Francis Chollet. This is the person that invented the benchmark. And basically he's talking about how, how LLMs are trained on massive troves of text, mostly pulled from the Internet. So it's likely that the same questions are being used to evaluate a model and were included in the training data. This is essentially what we call contamination when think about it like this. If you're going to train a model based on all of the text on the Internet, trying to get a completely new question is going to be pretty difficult considering the range of questions that exist in the current benchmark. So you can see right here, it says at best this is tipping the scales and at worst it's allowing them to simply regurgitate answers than rather performing any sort of human like reasoning, which is based on stuff that you completely haven't seen before. Like, imagine taking a test and you've already seen the answers. Like, you know, it's not taking a test, it's just pure memorization. You can see. And because AI developers typically don't really release details on their training data, you know, to those outside the companies, the people trying to prepare for the imminent arrival of AGI don't know if this is, like I said already, data contamination or this is affecting results. Okay, there are some research results that do state that these benchmarks fall dramatically when slightly reworded or they're entirely after the cutoff date of the training data. So that is an issue in some cases. And Francis Soleil, currently, this is his belief, he says that all current AI benchmarks can be solved purely via memorization. Memorization is useful, but intelligence is something else. And in the words of Jean, I honestly don't know how to say his surname.

Apologies.

Didn't want to completely butcher it. He says that kind of intelligence is what you do when you don't know what to Do. It's how you learn in the face of new circumstances, how you improve and adapt, and how you pick up new skills. And of course, this is the kind of reasoning that we want to see. And of course, this is where you can see, in 2019, Chollet published a paper in which he describes deceptively simple benchmark for evaluating AIs. And this is the abstraction and reasoning corpus, which is the ARC benchmark. And of course, you can see here, okay, by June 2024, this is where we get to the juicy bit. Okay, so by June 2020 had increased to 34%. Okay? And initially the best AIs could only solve 20% of our tasks. But by June 2024, this is only two months ago or a month and a half ago, that was at 34%, which is short of the 84%. But this is what I'm saying, when they wrote this article a few days ago, they were like, okay, this is only currently at 34%, but today we've got to jump up to 46%. Okay? And you can see here, and the reason I'm talking about this is because once this benchmark does get to around 85%, that means that whatever method is going to be used, that method is most likely going to be scaled up on top of existing LLMs and then used to reason. Basically what this does is it focuses these current models away from just scale and compute, which has been previously how we've managed to gain ground on current benchmarks. But what this tries to focus on is your reasoning techniques, for example, things like chain of thought, things like neurosymbolic AI, such as tool use and things like that. So you can see right here, he basically says that OpenAI basically set us back progress to AGI by 5 to 10 years. And the reason he says that is because focusing on LLMs like LLMs is a dead end. That's basically what he's saying.

You can see is the purpose of.

The ARK Prize is to redirect more AI research focus towards architectures that might lead us towards AGI, which is essentially what we want to do. And of course he says right here, the LLMs have essentially sucked the oxygen out of the room. Everyone is doing LLMs and of course, you know, LLMs is not going to lead to AGI, but I do think that, you know, LLMs are still an important part of AGI. Like I still think it's important, you know, part of the entire thing. But this is something that I think is really important because a jump up from, you know, like, like, like. I do think this is important because a jump up is insane, especially on this benchmark. And this is something that anyone can really do. And I will be intrigued to see if, you know, public companies are going to be trying to, you know, beat these benchmarks. You know, companies like OpenAI, companies like Goog. I do want to see if they can crush these benchmarks. And you can see here that it says leading AGI research lab DeepMind is implementing nearly identical techniques we're seeing at the top of arkprize leaderboards which is test time fine tuning and blast inference and search. So of course alpha proof, which was closed source and basically what they're saying here is that in contrast natural language based approach can hallucinate plausible but incorrect intermediate reasoning steps and solutions despite having access to orders of magnitude more data to we establish a bridge between these.

And this was essentially the Mathematic Olympiad.

Thing where Google managed to get a silver model and this is Google's alpha.

Proof where they managed to get a.

Silver medal at the Mathematical Olympiad. So that was remarkably impressive. But the point here is that they are using methods that are on the ArkPrize leaderboard. It's not like they're looking at the Arkprise leaderboard and using those methods. The point is that they're using similar approaches. They're not looking at just scaling up LLMs, just throwing money at it. They're looking. And the point that I'm trying to make here guys, is that when you have people actually looking in the direction, which is it might not be test time fine tuning and it might not just be search, but those are things.

That we've already seen has led to.

Superintelligence in other areas like AlphaGo. The point here is that this is actually like AGI coming through with a different level of reasoning. And the point here is that it's not just asking an LLM chain of prompt. This is a different way of thinking about the problem and this is really what's going to lead us to the vast majority of progress in the space.

Now with that really important news and like I said before, it didn't even get that much coverage. But I think it's the right direction in terms of actually getting to AGI. Francis Olay actually did say some stuff.

About this earlier this year, but basically.

He does say that solving ARC isn't.

Equivalent to solving AGI. The first arc solver isn't going to.

Be an AGI, but he's basically saying.

Here that until we solve ARC we.

Won'T definitively have AGI, since AIs cannot simply adapt to tasks they haven't seen before. And basically solving this benchmark is going to figure out how to make systems.

Actually adapt to on the fly, novel tasks.

And this is going to be the.

Major milestone on the way to AGI.

So that's why this is so important. Now, with all of this talk about AGI and all of those things, this brings me to something else that happened this week.

There was this AGI day and this was an event where there were loads of AI speakers speaking about the future of AI, what they thought.

And I wanted to address this video because there is a lot of, you know, debate going on right now about.

Where the future of AI is and.

Where things are currently headed. So there was a recent video. Gary Marcus spoke and he said that he's presenting evidence that the scaling of AI capability has slowed and we haven't seen any significant improvement in AI since models since GPT4 was trained in August of 2022. So I disagree with this and I'm going to explain why and why the future is about to get really crazy. But let's take a look at this. Okay?

They're all saying that there's room left for great improvements. Talk is cheap, but we haven't actually seen anything significantly better than GPT4. When was GPT4 actually trained as opposed to released? Turns out it was August of 2022. This has been well documented. They showed it to Bill Gates and kind of changed the world. But two years, we haven't really got much better. And so I'm going to show you a graph. I took the liberty of trying to extrapolate a curve because every day some AI influencer, AKA grifter. I didn't say that out loud, I just thought it shows you an exponential or I mean, doesn't show you an exponential. They're all math enumerate, but they say every day, Wow, I can't believe what came out this week. There's been exponential improvements. It's amazing. So I thought I'd actually plot a curve. I just did it by eye. I didn't do the math, but looking at palm chinchilla, which was the state of the art in August, sorry, April 2022 and relative, this is to release dates when GPT4 came out and you could genuinely make the argument that we saw exponential progress over that period, it was pretty amazing what happened. This is all since I spoke here three years ago. There was a lot of progress and you could argue GPT4 was kind of an outlier, but there was a lot of progress in the period leading up to GPT4. But is it continuing? So this is where the curve should go. There's a little glitch with that that we can talk about, but this is where the curve, roughly speaking, should go. But in reality, scaling has slowed. So here's the full curve of all the data. If you're a scientist, if you know Bayes theorem, if you know how to aggregate data, whatever statistical technique you want, you can look at this. Each data point is a test of a hypothesis. Let's say that scaling would continue at the pace that it did from palm chinchilla to GPT4. And it is obvious you don't even need to run a statistic, but you could if you like. It is obvious that scaling has slowed, that we are not in fact still on that exponential.

I'm not trying to be someone who's a doomer or someone that's trying to create any kind of debate, but I mean, there's simply one thing wrong with this entire graph is the fact that this is just simply basing this off.

If you guys can see on the.

Right hand side here, the mmlu, the five shot. And I mean, like, the thing is, is that as things improve, like for every percentage gain, there's not going to be, like, you can't exponentially gain from 87% to 90%.

Like, no matter what you do, even.

If you get to 100%, like, you could still argue that there isn't like a freaking, you know, exponential growth there. So I don't understand, you know, why Gary Marcus, as intelligent as he is, has as many, you know, accomplishments that he has. Waheed still just has this kind of perspective to where, you know, almost any kind of, you know, AI update, he only focuses on the bad things. Like, a lot of people who pay attention to AI updates, they say, oh, text video is amazing, but, you know, it doesn't get the fingers right. Oh, it gets the fingers right, but it gets this wrong. It's like you're just simply negating every other thing that it also does. Now, what I also want to say is that what we can also see here is that we can also see that yes, there was, you know, a huge jump from, you know, GPT 3.5 Turbo to GPT 4. And, you know, the plotting of this is, you know, wrong because number one, when we actually look at the actual dates of when things were actually released, I don't know why the MMOU is even Here, but this is wrong. For example, GPT 3.5, ChatGPT 3.5 was actually released late 2022, so that should actually be here. And then of course, GPT4 was released in March of 2022, which is March 2023, which is of course here. So you could say call, there's an exponential. Now, I'm not discounting other companies, but all of these other companies, these models, okay, like, you know, just these are first of all, of course, correctly, it does say open weight models here.

And I'm going to show you guys.

Another video from a new interview from Demis Hassabis talking about the future. But what we can see here is that these models like Gemini Ultra, Gemini 1.5 Pro, Llama 3, Claude 3, these models are a cycle behind. That's what you have to remember. Before the entire cycle, the only models that truly existed, that were truly competitive was chatgpt. There were levels ahead. And what you have to understand is that this makes sense. You can say scaling has slowed. If we get the next iteration of frontier models like Claw 3.5, Opus, Gemini 3, if we get those models and then we look and say, okay, there haven't been any new capabilities, the reasoning is still off, the hallucinations are still off, then yeah, you could say that. But right now we still haven't got that next wave of AI yet because Claude 2, the Claude 2 systems were still around GPT3 level and Claude 3 levels were just on power GPT4 level. So I would say this isn't scaling a son. I would say that if we're mid-2025 and the benchmarks haven't gone up, and I don't think that the MMO use a good benchmark anyways, because if you know anything about this benchmark, there's errors in this benchmark, which means that 100% is technically impossible if there's currently errors in the questions. So I think this benchmark isn't going to be a thing anyways. And right now Claude 3.5 Sonnet is currently the leader anyway. So I mean, once Claude 3.5 Opus gets here and other iterations of models, I just think that saying things like scaling has slowed. I mean, it doesn't really make sense because if you know anything, if anything, these companies are slowing down the releases. We got GPT 3.5, then six months later we got GPT 4. I mean, considering the fact that also we're now actually scaling up these systems in terms of hardware, in terms of the GPUs, you have to really think about, okay, there's going to be a longer time frame between these next iteration of models. And what we actually have seen is that yes, models are converging around this point, but this is all models converging.

Around the current point at their current size to reference.

And we've seen models get a lot more efficient, we've seen them get a lot faster, we've seen them get cheaper, we've seen context windows expand, we've seen reasoning increases. We've seen so many different things. I mean negating all of that saying scaling has so saying that it's an AI winter, saying the AI hype is dying down. I would just say always pay attention to the data because stuff like this is absolutely incredible. And I mean when you actually think about it, if we actually look at the time period from GPT 3.5, which is late 2022 to where we are now guys, that is literally late 2022 to where we are now, the guys, that's not even two years, it's not even two complete years. And people are saying scaling has slowed. Like look Back to when ChatGPT was released to where we are now, the state of the AI space. I mean it's absolutely incredible. So I mean saying that scaling has slowed is, you know, it's an incredible statement. But anyways, there was also something I wanted to show you as well, which is really important and this actually refers to what we discussed just a moment ago. But take a listen to Demis Hassabis.

I think that's the next era. These sort of more agent based systems we would call them or agentic systems that have agent like behav. But of course that's what we're expert in. That's what we used to build with all our game agents, AlphaGo and all of the other things we've talked about in the past. So a lot of what we're doing is bringing to kind of marrying that work that we're sort of, I guess, famous for and then with the new large multimodal models and I think that's going to be the next generation of systems. You can think of it as combining AlphaGo with Gemini.

Yeah, because I guess AlphaGo was very, very good at planning.

Yes, it was very good at planning. Of course only in the domain though of games and so we need to sort of general that into the, you know, the, the general domain of everyday workloads and language in 2, 3, 4 years time. Especially when you start getting agent like systems or agentic behaviors then I Think, you know, if it's something's misused by someone or perhaps even a rogue nation state, there could be serious harm.

Alphago those kind of things. Those things are going to be some of the most important areas. And what we've already seen, which is why I said this is really important news. Is that the way how Google's changing their approach to now focusing on those other systems that are getting them to really incredible benchmarks, like getting the Olympiad to silver medal, which is something that they didn't even think would happen. Many people predicting they were like, okay, that's going to be in 2026, at least to get a bronze medal or whatnot. And it's like a silver medal in 2024. They're like, yo, this is crazy. So them stating that, okay, we're now going to move to these methods, and these are the methods to that are moving on the ARC AGI benchmark, it kind of leads me to believe that if this is what information is publicly available, and considering the fact that OpenAI have historically been one to two years ahead of where the general consensus is of AI capabilities, it kind of leads me to believe that AGI is probably just around the corner. And there's also this one last thing that I'm going to leave you guys with, which is why I do think AI capabilities are probably still in terms of getting to AGI. It's probably still within a relative timeframe. And when I mean weak AGI, I don't mean like a complete AGI that can then develop super intelligence. I mean like an AGI that's going.

To be rather effective, I think probably.

Within the next two years. And the reason I say that is because when we actually look at OpenAI's future and the thing that they've said here, they've actually said that, like, if you look at this, this isn't actually an AGI chart. It just this. Like when I first looked at this, this is like, you know, I didn't really take it in because this is just stages of artificial intelligence. This isn't like AGI. Like, when we actually take into account what this is, this is, okay, level one chatbots. This is two reasons. Number three is agents, number four is innovators, and number five is organizations. When you actually really think about what's going on here, you could argue that level five, you know, that's going to be like AGI plus or near like ASI level. I mean, you know, an AGI system that can literally do the work on an entire organization. I Mean, if you think about what Apple and these companies are doing, they've created trillions of dollars in value, like millions and millions and billions of dollars. If you think about being able to have an AI system that can literally do the work of an organization as big as Microsoft, that's a huge level. Okay? And if you think about innovators, the AI that can aid in invention, this is like AGI asi.

Okay?

And if you think the fact that, okay, let's now plot this. The fact that, okay, arguably OpenAI is at this level, they basically recently kind of confirmed that they were at this level. I mean, you would say that getting to agent system, systems that can take actions, considering the fact that we've discussed that this can be solved, solving actions can be solved with scale, that I don't think it's going to take two whole years to get to this level. So I mean, I don't know about you guys, but I think that when we talk about AI that can aid in invention, that's like strong AGI bordering on asi. And this is definitely really strong AGI, ASI that can do inventions and then agents that can take actions. This is just going to be AGI with really good long term planning. So the reason is here is probably going to be weak AGI which is not far away what we're from. So I mean, it's going to be kind of interesting to see what happens in the future. But I do not think this space is slowing down by any means necessary. If anything, I would say things are speeding up. The only problem here is that a lot of companies are now private with what they're doing. So I think that with these levels and stuff like that, I do wonder if we're going to get, you know, notices and that stuff because we're dealing with really powerful technology here. But let me know what you guys think about this. What do you think the most, you know, important piece of AI news is? I'm going to be doing another video later. So, you know, subscribe for that and if you did enjoy the video, I'll see you in the next one.