Open AI O3 Models - Did Sam Deliver AGI for Christmas?

Hello everybody. Welcome back to another video here on the Matvid Pro AI YouTube channel. This is going to be my last video before the start of the holiday Christmas season. So to all of you, I wish you a very merry Christmas and if you don't celebrate Christmas, well, Happy Holidays. So three days ago we got our final announcement from OpenAI. This is the last day of the 12 days of OpenAI and literally all it was was an information Dr. We didn't get access to anything but a new top tier model and its information was shared with us from OpenAI. Interestingly enough, three days later we don't have a blog post from OpenAI. All that was made available was this OpenAI livestream that is 20 minutes long, but it's full of benchmarks and stuff and information, so plenty to go through. As you can see, it is OpenAI 03 and 03 mini. Obviously this is in the family of reasoning models, right? The O1 series, which we have access to now in chat GPT. Here's the thing though, OpenAI. What happened to O2? All the models we have access to right now are 01. So why did we jump to 03? Maybe the model is just that good. It can be deemed O3. Anyways, let's actually take a look at the live stream.

We view this as sort of the beginning of the next phase of AI where you can use these models to do increasingly complex tasks. They require a lot of reasoning. And so for the last day of this event, we thought it would be fun to go from one frontier model to our next frontier model. Today we're going to talk about that next frontier model, which you would think logically maybe should be called O2.

That's what I was saying.

But out of respect to our friends at Telefonica and in the grand tradition of OpenAI being really, truly bad at names, it's going to be called O3. Actually, we're not going to launch. Not launch. We're going to announce two models today, O3 and O3. Many. O3 is a very, very smart model. O3 mini is an incredibly smart model, but still but really good at performance and cost. So to get the bad news out of the way first, we're not going to publicly launch these today. The good news is we're going to make them available for public safety testing starting today.

All right, let's let the engineer talk.

And I want to talk a little bit about O3's capabilities. Now, O3 is a really strong model at very hard technical benchmarks and I'M going to start with coding benchmarks, if you could bring those up. So on software style benchmarks, we have Suite Bench Verified, which is a benchmark consisting of real world software tasks. We're seeing that O3 performs at about 71.7% accuracy, which is over 20% better than our O1 models. Now this pretty significant.

That's pretty dang big of a leap, honestly. 71 versus a 48.9 for the regular 01 that we just got access to. And then preview is all the way up. 41 competition code, we see even bigger of an uplift. Man, that's actually quite a huge jump.

I think there's one guy at OpenAI who's still like a 3000 something.

Yeah.

A few more months to enjoy.

Hopefully we have a couple months to enjoy there.

Great.

I mean, this is, it's. This model is incredible.

Programming.

Yeah, and not just programming, but also mathematics.

So dang. That's crazy. Okay, that's pretty wild. I don't know if you just picked up on what they said, but they said that an OpenAI employee at the top level for competition code is about Elo 3000 and this is just below that, which is wild. Man, that's definitely starting to get up there in these benchmarks. Right. Professional human level, top of the food chain. Dare I say there's been a lot of hype around this new O3 model since its release or announcement three days ago that this is AGI, artificial general intelligence. And I'm gonna talk about that towards the end of the video, but whether you want to call it AGI or not, there's no doubt in my mind that this clearly a very strong model if the benchmarks are to be believed.

So 03 gets about 96.7% accuracy versus an 01 performance of 83.3% on the Amy.

What's your best AMY score?

I did get a perfect score once. All right, so I'm safe.

Okay. He's still better. But this is an almost perfect score on competition math and that's really surprising. I mean, this is a huge leap. We're going from F territory to B minus territory to finally a territory. If you wanted to look at it from a grade point of view, that's a pretty big leap in understanding almost a perfect accuracy. And we go to PhD level science questions, GPQA diamond. We're all the way up to 87, which isn't as big of a bump in comparison to math. To be fair, there was a lot more ground to be made up in the Math sector. But yeah, they keep getting better.

And I think a couple have emerged as fairly promising over the last months. One in particular I want to call out is EPIC AI's frontier math benchmark. Now you can see the scores look a lot lower than they did for the the previous benchmarks we showed. And this is because this is considered today the toughest mathematical benchmark out there. This is a data set that consists of novel, unpublished, and also very hard.

These are extremely hard.

Yeah, very, very hard problems. Even turns Taosis. You know, it would take professional mathematicians hours or even days to solve one of these problems. And today, all offerings out there have less than 2% accuracy on this benchmark. And we're seeing with O3 in aggressive test time settings, we're able to get over 25%.

Wow. Take a look at this one right here. So this is the Arc AGI semi private eval, obviously another benchmark. The original 01 series look mini is all the way down here at 8%, preview at 13%, low amounts of compute at 25, 31% for medium amounts of compute, and then 32% for high amounts of compute. You can see there's a curve here where we're getting diminishing results as we throw more compute at the O1 series of models. But with O3 series, low compute is way, way better than even the high compute on O1. And the leveling off curve is just in a different stratosphere. I mean, with low compute, we're seeing twice the score. And even more than that with high compute at 87.5% completion here. That's pretty crazy, man. These, these benchmar, if they aren't lying, this model is stonkingly good. Now we're jumping into some more benchmarks that now mention the O3 mini series of models, which is supposed to be more efficient for less difficult tasks. As you can see, the cost for the O3 series of models is more efficient across the board. But specifically in the MINI models, we get nice high Elo scores, decent Elo scores, definitely better than 01 Mini and regular 01, even at a much, much cheaper cost overall. Regular O3, though, is still very much expensive. Let's get to them actually demoing these models and using them.

So I'm testing out O3 mini high, and the task is that I'm asking the model to use Python to implement a code generator and executor. So if I launch this, run this, like Python script, it will launch a server with a UI that contains a text box and then we can make coding requests in a text box. It will send the Request to call O3 Mini API and O3 Mini API will solve the task and return a piece of code. And you will then save the code locally on my desktop and then open a terminal to execute the code automatically.

So it's a very complicated task.

Pretty complicated task. Right.

Complex.

A big chunk of code. So if we copy the code and paste it to our server and then we would like to run launch this server.

So we should get a text box when you're launching it.

Yeah.

Okay, great. Oh yeah, I hope so.

It seems to be launching something.

Oh great.

We have a UI where we can.

Enter what he said it was gonna do.

Simple one like train OpenAI and random number. So it's sending the request to O3 mini medium. So it should be pretty fast. Right. So on this terminal, yeah, 41, that's the magic number. Right. So it saves the generated code to this like local script on a desktop and the print out opening end 41.

Man, that is wild. Like, look at this prompt. Would you even know where to start? Most of us probably wouldn't. The prompt isn't especially well devised either. It looks a little thrown together. I mean we've got like some basic text at the top for what it needs to do and then we've got some more key details and that's it. That's all we send off to this thing. And it's able to something pretty dang complicated. Man. I will say these are some pretty interesting benchmarks right here though. We have internal function calling with structured outputs. Not a huge increase in accuracy between all of these models. To be honest, it looks like we're kind of leveling off at least for this specific benchmark. With a pretty high success rate, no doubt for all of these models, but the highest, technically speaking, with O3 mini high with live bench coding, we can see that original 01 is handedly better than both O3 mini low, O3 mini medium, but not better than O3 mini high. And again, O3 mini high is a lot cheaper to run. And wow, the internal function calling again without the structured outputs, it's very much leveled off. I mean overall though we look at the PhD level science questions. GPQA Diamond, 01 Mini and 03 Mini Low are pretty much in line. 03 Mini Medium and O3 Mini High still don't beat both O1 Preview and O1 with the only model to really talk everybody in this being big bad 03. So it seems like some of these benchmarks are requiring those larger models. Some of these mini models, they just can't output the higher scores on these benchmarks. The parameter count is just too low. Is that the answer? Is that hypothesis? This is probably more of a hypothesis that the researchers would be working on. But that's what, that's what I'm wondering about, looking at this. So, yeah, if you want to test the O3 model and get access to it, you're gonna have to fill out this form first. Last name, institution or organizational affiliation, your account, etc. They're probably going to be bringing people that are more trusted in on this, especially since they just had a SORA leak not that long ago. I think getting access to this model is going to be pretty difficult. Probably around summertime of next year, I Predict, is when O3 mini will be available to the public. All right, so now that we took a good look at the announcement itself, there are certain extrapolations that we can make. But before that, I have a quick word from today's sponsor. Today's video is sponsored by fetchfox. Fetchbox is a Chrome extension that can scrape any data from any website using AI. The best part is that it's really easy to use. Simply install the Chrome extension right from the Chrome web store and then on any webpage, simply click on the Extensions tab in Chrome and then click on Fetch Fox. From here, I can prompt Fetch Fox as if it was a person. As you can see, we're on the SpongeBob SquarePants Wikipedia page. Give me a list of people who work worked on the show and we'll send it off to Fetch Fox. Then Fetch Fox uses AI to generate a scraping plan. You can see in this case it's starting on the Spongebob Wikipedia page, but you could start on multiple URLs and then it specifically devises a plan to extract the data you're looking for for this simple prompt. It's the name of a person who worked on the show and what their role was. Then all we have to do is click, run, scrape, and as you can see, we actually have some names filling in here, obviously. Stefan Hillenburg, the creat, Derek Creative Director, Tim Hill, writer. We've got voice actors in here. It's grabbing all the names and roles exactly as I asked. And once it's all done, we can go down to the bottom here and download it as a CSV. Now, guys, keep in mind, because FetchBox is powered by AI, it could do a lot more complicated scraping tasks than just this. The fact, though, that it's an all in one web scraper and AI powered data curator makes this thing a no brainer to have as a chrome extension. Fetch fo AI allows for up to 1000 free items per month scraped and after that a pay as you go option for a dollar per 1,000 items scraped. They've also got enterprise plans as well, and if you're not a fan of their plans, you can actually bring your own OpenAI key right directly in Fetchfox. These are all, in my opinion, great options at a good value for the consumer and I especially love that you can give it a good test run for free. Click the link down below to try Fetch Fox for free. It works great and I'm a big fan. Now back to your regularly scheduled content. Welcome back everybody. Thank you so much for sitting through and dealing with sponsors. They are absolutely the reason that I can pursue this channel full time. But not the only reason, of course. You guys watching make this a reality for me, and I am forever thankful of that. Regardless, let's talk about O3. So it's a big model, right? Next step in the Reasoning phase from OpenAI. Tearing up benchmarks. Really difficult benchmarks too, by the way. Ones that you wouldn't be able to solve, most likely. I mean, not all of you, maybe some of you are geniuses. But can a single one of you solve all of the benchmarks that OpenAI03 is solving at that Elo score? No. It's turning into super intelligence. A super genius in a lot of ways. Even the older ChatGPT models are super geniuses. They can hold more information than your could ever hope to. At any rate, a lot of the commotion that I saw online was talking about whether or not O3 was AGI. So sit down, let's have the AGI talk. Let's go through this. First of all, what is AGI and how do we define it? Well, it stands for Artificial General intelligence. But here's the thing you need to know. It's a theoretical concept. It's loose. The definition for AGI kind of depends on the person who is observing. I might have a different definition than you do, because guess what? It's not in the dictionary. It's not really a real term in that sense. There's a Wikipedia page about it. There's a lot of talk online about it, but no one truly knows what it is because it's not set in stone. There's nowhere you can really point to and say, hey, that's definitely AGI. We know AGI. Is this a Quick Google search and AI Labs overview will say that AGI refers to a hypothetical type of intelligence and in a machine that would allow it to perform any intellectual task a human can, essentially mimicking a human level cognitive ability, learning, reasoning, adapting across various situations. And it specifically says unlike current AI systems, can O3 perform any intellectual task a human can? Well, the answer is immediately no, because humans are still better on some of these benchmarks. Can one single human, you know, perfect all of these benchmarks? Well, no, but again, these are the things that we have to think about when we're trying to define AGI. The model is absolutely reasoning, I think there's no doubt about that. But can it learn well over time? I mean if you count the training, yes. It's also probably doing some minor level of learning during reasoning. What's the difference between reasoning and learning? For example, it's, it's very loose, it's very wishy washy. That's the point I'm trying to make. Is OpenAI's O3 AGI? Well, what's your definition of AGI? That's really what it comes down to. Is it a new top charting model? That is pretty mind blowing? I would say absolutely. I don't think the benchmarks that OpenAI are presenting us are necessarily skewed or cherry picked or weaved in a certain way. I think they're probably pretty accurate and it's clear that they're making great headway in their reasoning models. Would I personally say, hey, this is AGI? No, it's not better than a human at everything. It's not able to actually go off and do advanced tool use or computer use, for example. These models aren't plugged into the real world even. I think for me a good test for whether a model is AGI or not would be the ability to give it a controller, you know, like a regular Xbox controller. Hook that up to some sort of a robot with a camera that takes an input, give it, give it all those inputs and give it the ability to output to the controller and see if it can freakin control the robot. It's that leap from text to the real world, I think that is going to make the biggest difference. Once you can directly link the two flawlessly, I will be very, very impressed because that means that all of the text knowledge, crystallization in the form of text that these models are perfecting, right? The art of perfecting them by mastering these benchmarks, once that translates directly into reasonable actions in the human world, the ability for one of these large Language models to control a robot to fold your laundry or something like that. That will be wild to me. I hope I explained that concept of mine well enough. Like obviously there are laundry folding robots that already exist. I've seen videos of them. But they were specifically trained to fold laundry. Artificial general intelligence, meaning generalize across these tasks. Right. Without specifically being trained to do so. That's what would impress me. Alright, so now that that's over, I kind of want to give my thoughts on where we're headed. Especially since it's the end of the year we're closing out. The 12 days of OpenAI was overall of a mixed bag. I think 03 here was probably one of the cooler things that we've seen. Looks like really impressive models. Hopefully they're still multimodal. I assume they are since they're in that o series like GPT4O, which stands for Omni. A lot of the releases we saw from OpenAI I didn't even deeply cover because they were more just like little developer tweaks or updates for chat GPT that weren't huge. OpenAI also announced Sora, but then was quickly overshadowed by Google's announcement of their killer VO2AI video generator, which is beyond anything I've ever seen before. If there's any takeaway. Honestly, coming up to the end of 2024, there is no sign of any of this slowing down. More compute is coming from the likes of Nvidia and other companies. The models are still getting better with new paradigms like reasoning. The competition is still very much fierce. It's not just OpenAI dominating everyone. Well, maybe in the text area. O3 is unlike any other model we've seen. But Google, Anthropic, the rest of them keep coming back and slapping down with their own models that are highly competitive. And plus we're also seeing unprecedented growth in the open source area. Open source models are better than ever and the updates are more rapid than ever. Everyone is keeping up with each other and it's a very, very tight competitive space. All I can say is at the end of the day, that's good for people like you and me. We want this technology to be put in the hands of everybody. It can't just be in the hands of celebrity, few people. It's extremely powerful technology. We've talked about this before. I mean, you see what O3 is doing. It's doing tasks that most of us don't know how to even begin to start. Can it walk you through how to complete other tasks? Hopefully better than the Other models as well. This is the beginning of super intelligence and immediate access to information for everybody, regardless of who you are. It's very powerful. I do feel though, at this point, especially with the text models, it's getting a little bit more difficult to be increasingly more excited for the next one because they don't have things like direct access to your computer, they don't have things like direct access to your phone. Not that everyone wants them to have access, but they would be much more powerful if they did. And at this point, right, like we're seeing him complete tasks that we can't complete ourselves and a lot of us don't even understand and then all we really have to go off of is benchmarks and charts, which are a little bit, bit boring, I suppose. Again, I think the excitement lies within the space as a whole. You can see the competition, you can see the prices coming down, you can see more people get access to smarter and smarter models. And at the end of the day, what's truly exciting is what people are going to do with the models, how they're going to use them to increase their productivity, their creativity. I think you're going to see a whole new way of navigating life for a lot of people because all of this knowledge is going to be immediately accessible. We already had that with Google to a wide degree, but now it's custom tailored for just you and can be directly applied into your life. Smooth sailing. I mean it's, it's wild, right? Anyways, in general, the AI space has quieted down a little bit for the holiday season as I mentioned right in the beginning of this video. So this was sort of the final punch I think at the end of this year coming from OpenAI with their, their new O3, the benchmarks, best text model we've ever seen. Still so much other exciting stuff going on to. I'm really excited for 2025 guys. I don't know about you. I'm ready for the AI models to take control of my computer and start setting up things for me that I never even knew were possible. Taking care of problems while I'm away doing other things. I'm excited to create really good looking quality AI videos and actually begin to tell stories using them. I think the quality is reaching that level where in a lot of shots you just won't be able to tell that it was AI generated in the first place. And of course, don't forget about AI music, which is still very, very much a competitive space that is being worked on. And the non stop barrage of awesome quality open source AI models that continuously seems to never stop. What excites you the most in 2025 in regards to AI tech? Do you think we're going to start seeing a hundred percent on certain benchmarks and do you think your definition of AGI have we already achieved it? Are we we going to achieve it in 2025 if not and finally probably the biggest question what is our world going to look like in the future with all of this AI technology integrated seamlessly? That one has been racking my mind lately and I can't imagine what kinds of things are going to be possible and what creative people are going to do with the power of AI. Alright, I'll see you guys in the next video. Thank you so much for watching and supporting me throughout this entire year. Like I said, I think I am going to have some other updates before New Year's but enjoy your holidays. Have a very merry Christmas and folks I will see you in the next one. Goodbye.