OpenAI Just Revealed They ACHIEVED AGI (OpenAI o3 Explained)

So today actually marks a very historic moment for the AI community as it is going to be probably regarded as the day where AGI actually happened. Now, if you guys don't know why this is the case, this is because OpenAI today released, slash, announced, I guess you could say their new O3 model, which is the second iteration of their O1 series, the model that thinks for a long time. Now, if you don't understand why this is potentially AGI, this is because the new system is managed to surpass human performance in the ARC benchmark. Now, the reason that the ARC benchmark is such an important benchmark is because it is resistant to memorization.

Sure. So ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from most LLM benchmarks out there is that it's designed to be resistant to memorization. So if you look at the way LLMs work, they are basically this big interpolative memory. And the way you scale up their capabilities is by trying to cram as much knowledge and patterns as possible into them. And by contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as core knowledge, which is basic knowledge about things like elementary physics, objectness, counting, that sort of thing. The sort of knowledge that any four year old or five year old possesses. Right. But what's interesting is that each puzzle in ARK is novel, is something that you've probably not encountered before, even if you've memorized the entire Internet.

Now, if you want to know what ARK actually looks like in terms of this test that humans are so easily able to pass, but you know, these AI systems currently aren't. You can take a look at the current examples right here in this video example here.

AGI is all about having input examples and output examples. Well, they're good, they're good. Okay. Input examples and output examples. Now, the goal is you want to understand the rule of the transformation and guess it on the output. So Sam, what do you think is happening in here?

Probably putting a dark blue square in the empty space.

See? Yes, that is exactly it. Now that is really. It's easy for humans to intuitively guess what that is. It's actually surprisingly hard for AI to know, to understand what's going on. What's interesting though is AI has not been able to get this problem thus far. And even though that we verified that a panel of humans could actually do it. Now, the unique part about ARC AGI is every task requires distinct skills. And what I mean by that is we won't ask, there won't be another task that you need to fill in the corners of blue squares and but we do that on purpose. And the reason why we do that is because we want to test the model's ability to learn new skills on the fly. We don't just want it to repeat what it's already memorized. That's the whole point here. Now, ARC AGI version 1 took five years to go from 0% to 5% with leading frontier models. However, today I'm very excited to say that O3 has scored a new state of the art score that we have verified on low compute for O3. It has scored 75.7 on ARC AGI's semi private whole holdout set. Now this is extremely impressive because this is within the compute requirements that we have for our public leaderboard. And this is the new number one entry on RKGI Pub. So congratulations.

Now I know those of you outside the AI community might not think this is a big deal, but this is a really big deal because this is something that we've been trying to solve for I think around five years. Now. This is a benchmark that many would have heralded to be the golden standard for AI and would of course mark the first time that we've actually managed to get to a system that can, that can actually outperform humans at a task that traditionally AI systems would particularly fail at. Now what was interesting was that they had two versions. So we had O3 with low tuning and we had O3 with high tuning. So the O3 with low tuning is the low reasoning effort. And this is the model that operates with minimum computational effort, which is optimized for speed and cost efficiency. And this one is suitable for simpler tasks where deep reasoning is not required. So basic coding, straightforward tasks. And then of course you have the high tuned one, which is where the model takes more time and resources to analyze and solve problems. And this is optimized for performance on complex tasks requiring deeper reasoning or multi step problem solving. And what we can see here is that when we actually, you know, tune the model and make it think for longer, we can see that we actually managed to surpass where humans currently are. Now what's crazy about this is that the people that have created the ARC AGI benchmark said that the performance on ARC AGI highlights a genuine breakthrough in novelty adaptation. This is not incremental progress. We are in new territory. So they start to ask, is this AGI? O3 still fails on some very easy tasks, indicating fundamental differences with human intelligence. And the guy that you saw at the beginning, Francis Chollet, actually spoke about how he doesn't believe that this is exactly AGI, but it does represent a big milestone on the way towards AGI. He says there's still a fair number of easy arc AGI1 tasks that O3 can't solve. And we have early indications that arc AGI2 will, will remain extremely challenging for O3. So he's stating that it shows that it's feasible to create unsaturated interesting benchmarks that are easy for humans, yet impossible for AI without involving specialist knowledge. And he states now, which is, you know, some people could argue that he's moved the goal that we will have AGI when creating such evals become outright impossible. But this is a little bit contrast to what he said earlier this year. Take a look at what he said six months ago about the benchmark surpassing 80%, which it did today. Turn the question around to you. So suppose that it's the case that.

In a year a multimodal model can.

Solve ARC, let's say get 80% whatever the average human would get than AGI?

Quite possibly, yes. I think if you start. So honestly what I would like to see is an LLM type model solving ARC at like 80%. But after having only been trained on core knowledge related stuff, now one of.

The limitations of the model is actually the compute cost. So you can see right here that he says, does this mean that the ARK price competition is beaten? He says no. The ARK price competition targets the fully private dataset, which is a different and somewhat harder evaluation. But the ARK prize is of course the one where your solutions must run with a fixed amount of compute, which is about 10 cents per task. And the reason that that is really interesting is because I'm not sure you guys may have seen this, but if we actually take a look down the bottom here, you can see that O3 high tuned the amount of compute that is being put into the model there. And means that this model costs around $1,000 per task, which is pretty, pretty expensive if you're trying to use the AI for anything at all. I think these models that search over many different solutions are going to be really, really expensive. And we can see that reflected here with this one being over $1,000 per task, which is ridiculously expensive when you think about using it to perform any kind of task. Of course, as we've seen with AI cost will eventually come down. So the fact that they've managed to get the benchmark is just the very best thing we can use. The similarities between how technology was pretty bulky in the early days, you know, like the big TVs, the really bulky phones, but now with technology you find ways to become more and more efficient and eventually you do things faster and of course cheaper. So it's quite likely that this will happen in AI too. And what I do find crazy about this is that, you know, two years ago he said that the ARC AGI benchmark, fully solved, is not going to be within the next eight years. And he says 70% hook, hopefully less than eight years, perhaps four or five. Of course, you can see AI managing to speed past most people's predictions. We also got the fact that it did very well on the SWE bench, which is of course a very hard software engineering bench. And I'm guessing that if you are a software engineer, this is probably not the best news for you. But I'm sure that now, with a bunch of people who are now coding, there's probably a lot more demand for software engineers that actually understand the code that's actually being written here. But this is actually rather interesting because one thing that I realized when doing this video was, was the fact that this is actually O2 and not O3. Just want to go on a tangent quickly here because whilst I was reading this breakthrough about O3, one thing that I needed to remember is that this is actually the second iteration of the model. I think some people might be thinking that O3 is OpenAI's third iteration, but O2 is simply being skipped because there is a conflict with O2, which is a British mobile service provider. So the fact that this is only the second iteration of the model, it does go to show that potentially 03 or even 04 is going to be quite a large jump and might reach benchmark saturation. Now we also look at the math benchmarks and the PhD science level benchmarks, we can see that there is a decent improvement there as well. I do think though that this is sort of reaching the benchmark saturation area because we can see this one on competition, Math is around 96.7% and this one is 87.7%. And like I said before, one of the things that most people are starting to say is that like, okay, AI has slowed down because because it's no longer increasing at the same rate as it was before. We have to understand is that as these benchmarks get to 95 plus or around that 89%, the incremental gains are going to be harder and harder to reach, because number one, you only have 10% left to get. And number two, it's quite likely that 3 to 5% of all the questions are contaminated, meaning that there are errors in those questions anyways, which means that 100% is simply not possible on certain benchmarks. Which is why they decided to create the Frontier Math benchmark. Now, as at the time when the benchmark was released, which is I think around two or three months ago, these were the current models that could do only 2% on these questions. You can see at the lead was Gemini 1.5 Pro Claude 3 Sonnet. And you can see O1 preview and 01 mini were there. But for those of you that haven't realized just how good O2.03 is, this is a model that gets 25%. So this is something that is really, really incredible when it comes to research math. And you have to understand that this kind of math is super, super difficult. And all of those questions are completely novel. So the previous benchmarks that we aren't truly showing the capabilities. I mean, think about it like this, okay? When we take a look at these benchmarks, you can see that, okay, you would say maybe the model's gotten 10% better overall, which is just not true. Because if we actually look at the really hard benchmarks where it's really solving unseen math problems, we can see that there is a 20 times improvement over the current state of the art, which is just absolutely incredible. So I think this is probably the very important image that people need to realize because you can't compare this kind of model, even though it does do a massive increase on these kind of benchmarks, but you can't really compare it to this where it's having this insane level of scores. Now what's important to understand is that Noah Brown, the person who works on reasoning at OpenAI, said that O3 is going to continue in that trajectory. So for those of you who are thinking maybe AI is slowing down, it is clear that that is not the case. Now. Interestingly enough, we also got Sam Altman in an interview talking about what he believes AGI to be. And I think like I said before, the definition is constantly shifting and constantly changing.

It used to be a term that people used a lot and it was this really smart AI that was very far off. In the future, as we get closer to it, I think it's become a less useful term. People use it to mean very different things. Some people use it to mean something that's not that different than.01, you know, and some People use it to mean true super intelligence. Something smarter than like all of humanity put together. We try now to use these different levels. We have a five level framework now. We're on level two with agents now. Sorry with reasoning now rather than the binary of is it AGI or is it not? I think that became to course as we get closer but I will say by the end of next year, end of 25, I expect we will have systems that can do truly astonishing cognitive tasks like where you'll use it and be like that thing is smarter than me at a lot of, a lot of hard problems.

So now I can see the only thing here is that if you are a safety researcher, he says please consider applying to help test O3 minion O3 because he's excited to get these out for general availability soon and he's extremely proud of the work that OpenAI have been doing to creating these these amazing models. So it will be interesting to see when these models are actually released, what people are going to do with them, what they're going to build and of course, what happens next. If you enjoyed this video, let me know what your thoughts are on if this is AGI or not. I do think that that benchmark has been broken and we're constantly finding new ways, so it will be interesting to see where things head next.