The King is Back. o3 & o4-mini are ELECTRIC! Can Google Compete?

Wham. That is the sound of OpenAI dropping the hammer down on every single other AI company in the world. They released new models, new frameworks. We are going to dive deep into them today. Hey, but first things first here, right? What exactly are we talking about? Well, we've got 03 and 04 Mini as our two big models. We're mainly going to be discussing today. OpenAI claims here are smartest and most capable models to date with full tool access. And let's talk about rollout real quick because I'm sure a lot of you people already have ChatGPT subscriptions they will be on. Plus I already have access of course, Pro and then I guess Edu and teams members gotta wait like a week. But yeah, I already have access to 04 mini and O3 inside of my chat GPT. So go ahead and check yours. If you are on a paid plan, free users are still going to have to wait. For now, the brand new experience inside of ChatGPT with these new models is very intriguing to me. It feels agentic. It can do complicated math, it can search the web, it can make charts for you, it can run code. It seems really great at being able to solve problems in a realistic and scientific way. Like in this prompt, for example, we ask should a basketball move its corner three line further out? Run a thorough analysis of existing sentiment with projections and visualizations of the impact of such a move and make a recommendation. And this is on O3 here. So the first thing it does of course is thinking. These are thinking models. So it's going to analyze the prompt, it's going to come up with a plan of action to solve for the question. It searches the web to get more data and then it's going to analyze. It's actually going to do some math, it's going to run some Python code and give us a graph even to come up with the best answer possible. It's really quite surprising and it's a lot of fun to use in practice. Final recommendation with a bottom line that's based on a lot of detailed research and analysis. All right, now let's talk about the differences between O3 and 04.03 here is state of the art. It's a larger model, literally more parameters, and it's meant to really push the front of all of the different AI capabilities. Coding, math, science, visual perception. OpenAI claims state of the art benchmarks, which we are going to take a look at pretty soon. Of course, I always say take the benchmarks from these companies with a little bit of a grain of salt. We have to wait for the vibes from the community to come in. Obviously this just released, so we don't have a lot of testing from the community in just yet, but I have a little bit to talk about. I'll probably be doing a follow up video next week that dives a little bit deeper into what the community has discovered is possible regarding these brand new models from Open Anyways, I found this particularly interesting in evaluations by external experts. O3 makes 20% fewer major errors than OpenAI's 01 on difficult real world tasks, especially excelling in areas like programming, business and consulting, and creative ideation. Early testers highlighted its analytical rigor as a thought partner and emphasized its ability to generate and critically evaluate novel hypotheses. This is a big deal. I think that AI has had this capability for a while, but it seems to be leveling up with each new model release. So hopefully now it's able to work along with the top humans in these fields for research and biology, math and engineering. So that's O3. It is the best. It's the biggest model. It's meant to really be the top charter, the heavy hitter for OpenAI. Now O4 mini, obviously let's talk about the naming real quick. Obviously this is a smaller model being named mini, but it is 04 which implies it's from the next generation after 03 and it kind of is. I know the name ish game is pretty confusing, but essentially O3 is the bigger, badder full model from this current generation. We finally got access to it. We've had access to O3 mini for a while. O4 mini is now replacing O3 mini and it's like a glimpse into the future from a smaller model. My thoughts is that the full 04 is probably not done training, it's not ready for prime time and well we just got access to 03, so obviously we're not going to have 04 just yet. Okay, so 04 mini is a smaller model. It's optimized for fast cost efficient reasoning. However, as OpenAI states and they claim here, it achieves remarkable performance for its size and cost. And again that's going to apply to the fields like math, code and visual tasks. Scrolling down to actually take a look at the benchmarks themselves, you'll notice that OpenAI is not comparing to any Google models, any open source models, anything like that. It's all against their own models. But don't worry, I have some charts of my own that compare these models directly to stuff like Google's Gemini 2.5. For now though, these are the charts that they present to us and they are crazy. First off, AIM 2024 and AIM 2025 original 01 was around the 70s in accuracy. 03 mini is a little bit better. 87, 86. Let's now just look at 03 with no tools at all. In AIM 2024 we're looking at a 91.6 and an 88.9 in 2025. Already up there. 04 mini with no tools is right behind it, stomping all over both 01 and 03 mini. 92.7 in AIM 2025 and 93.4 in 2024. Now let's add tool use because that's a big deal and that's a big part of what we're going to be talking about with these models today. If we add tools using Python web search, our scores go up even higher.03 with only python gets a 95 in aim 2024. And in 2025 that jumps up to. The crazy part is that 04 mini with python use actually beats out O3 in these benchmarks. It is the highest scoring model, a 98.7 and a 99.5, which is crazy. Clearly, when you give these models correct access to tool use and actually train them to use the tools, benchmark scores are going up and they're pretty much fully saturated, at least for AIM 2024 and 2025. Here we have reached the top. 99.5 is essentially a perfect score. And check out CodeForce's competition. Co those scores for 01 and 03 mini are all the way down here, barely touching around the 2000 mark. 03 with terminal and 04 with terminal dominate. 2706 for oh three and then a better score for 04 mini. 2719. Crazy. It's actually beating out 03 and that is a much smaller and cheaper model. But again, you know, these are the benchmarks right from OpenAI and there's other companies and other AI models out there, so we're going to be looking at other stuff in the future. So far, crazy benchmarks. All right, now, these benchmarks are a little bit more favored by me personally. I think that these are a little bit more fun to look at and a little bit more telling. All right, so for GPQA diamond, we see a slight lead from the new 03 and 04 Mini. But this is without tool use, just scoring a couple of points over the older models. But if we move over to humanity's last exam. This is where things really start to get intriguing. First up, lowest score 01 Pro gets like an 8.12, not a great score. Oh 3 mini does quite a bit better at a 13.4. And if we go over and look at 04 mini with no tools, we can see it can say 14.28. So only a slight edge about a percentage so for O4 mini versus O3 mini. But let's say we give 04 mini some tools, Python and browsing. Well, the score jumps up to a 17.7, which puts it pretty far ahead. However, O3 with no tools at all, that larger model destroys and decimates O4 mini and O3. And if we give O3 python and browsing and tools while that score jumps up to a 20, 24.9. Interestingly though, none of these scores can actually compete with OpenAI's very own deep Research, which we've had access to for a while. And from what I understand, Deep Research is powered by O3 in some regard under the hood, it's not the same O3 that we got today. I think it's a version of O3 specifically fine tuned for Deep Research. Either way, it tops the scores on humanity's last exam. But I imagine that is partially due to the fact that this model is fine tuned just for Deep Research in itself. So it's good to see that a more generalized three is catching up to a very similar score. And if we scroll through the rest of these benchmarks, the story really continues from there. Multimodal benchmarks definitely seems to be quite a bit better than the older models. Again, same thing goes for coding. Definitely better than OpenAI's older models. But how do they compare to like Gemini? Well, using 0.4Mini high itself in chat GPT, I was able to have it do some research for me, get the scores and actually create a new chart that compares it to Gemini 2.5 Pro, which is Google's current app Best model. I mean state of the art wipes the floor encoding. I mean it's the preferred coding model for a lot of people right now. So at a first glance, how does it look compared to these two new models? Well, in AIM 2024, Gemini 2.5 Pro is just behind both 04 Mini and O3. And AIM 2025, the story continues again just behind these two new models. So they're very close. They're very close. GPQA Diamond Gemini 2.5 does take the lead here. It actually wins, but it is so, so close that it' Essentially the same in humanity's last exam, though we can see that 04 mini handily loses to Gemini 2.5 pro. But the full O3 ekes it out just a little bit. And if you remember, deep research is somehow even better on this particular benchmark for mmmu. It's a little obscured here by the chart at the top, but you can see Gemini 2.5 and 04 mini get approximately the same score with 03 just eking it out just a tiny bit. Now this is performance and accuracy, right? How about actual pricing? Because that's very important here. Sure, you know, a model can be better in certain use cases, but if it's way more expensive, it's not going to be worth using. So let's start off here with input. This is tokens that you're inputting in your prompt to the model. The pricing here is a pretty dramatic difference.04 mini is a great model, all right. It's, it's cheap, it performs well, competes with 2.5 Pro. You can see 2.5 Pro is also a very cheap model in the API. O3, the full big bad 03 is way, way more expensive. It's not even close for the INP tokens. 10 bucks per million input tokens versus like a dollar something for both Gemini 2.5 Pro and 04 mini. And the story is very similar for output pricing. Gemini 2.5 Pro is about twice as expensive as 04 Mini, but still a lot cheaper than O3, which is 40 bucks per a million output tokens versus only 10 from Gemini 2.5 Pro and less than half of that from 04 Mini. In terms of cost effectiveness, it looks like 0.4Mini is going to be a pretty slapping model to compete with Gemini 2.5 Pro, at least in this regard. O3 is super expensive while barely being better than Gemini 2.5 Pro, at LEAST from the benchmarks we've seen so far. So let's talk a little bit about training these models. OpenAI claims that with O3 and 04 Mini, they've pushed an additional order of magnitude in both training compute and inference time reasoning, and they're still seeing clear performance gains. So they mean by how much it costs to run the model. They're still getting more performance gain by scaling both pre training and reinforcement learning. Here's the interesting part though. They've trained both O3 and 04 mini to use tools through reinforcement learning, teaching them not just how to use the tools but actually reason about them when they use them. Their ability to Deploy tools based on desired outcomes makes them more capable in open ended situations such as as a lot of those in ChatGPT. When you're just chatting away, asking all kinds of questions, diving deep into all kinds of topics, but particularly those involving visual reasoning and multi step workflows, you're going to see the most benefiting gain. Now let's talk about multimodality and inputting images in your prompts. Apparently for the first time, these models can integrate images directly into their chain of thought. They don't just see an image, they actually think with it. This unlocks a new class of problem solving that blends visual and textual reasoning together, reflected in their state of the art performance across multimodal benchmarks. You could upload a photo of a whiteboard, text, diagram or hand drawn sketch and the model can interpret it even if the image is blurry, reversed or low quality. With tool use, the models can manipulate the images on the fly to correct for them, rotate, zoom, transform them. So if there was some really small text on a whiteboard, it could zoom in, grab, read whatever that text is and then zoom back out and zoom into another part of the image. And this to me is a huge part about why these models feel so much more agentic than the previous ones. They feel automated. They use the tools that they need based on what it thinks it needs to do to solve whatever problem you give it. And it will think sometimes for you know, two, three minutes about a problem and then give you a response. But that response is of a quality that is a whole extra caliber higher than we're used to receiving from not only non reasoning models, but also previous reasoning models like O3 mini. They talk a little bit more about moving towards agentic tool use. These models are trained to reason about how to solve problems, choosing when and how to use tools to produce detailed and thoughtful answers in the right output formats quickly, typically in under a minute. Or they that's what they'll strive to do. Depends on your prompt. OpenAI has some examples on their blog post here comparing O3 to 01. Not like I could solve this mathematical problem anyways, but OpenAI would not be putting this on their blog blog post if it was incorrect. You can see that O3 clearly gets a correct answer at the bottom here where 01 kind of fails. Here's a pretty great example they're providing for reasoning throughout the images. You can see that it analyzes the initial one and then it's like oh God, we gotta zoom in here to get more detail so it does an extra little zoom in, and now it analyzes this particular image, gets the data, moves on to the next part of the image, so on and so forth. It can actually look and highlight specific areas and zoom in as if you were a person read through and looking at that piece of paper. Even highlighting specific columns and remaking them out in the chat, you know, to eventually give you your final answer. Reordering and reorganizing the entire itinerary to be more efficient in terms of cost efficiency. Here we can see that compared to OpenAI's older models, O4 mini is smarter than O3 mini while also being more cost effective. And the Same goes for O3 here. It's actually quite a bit more cost effective than even O4 mini is compared to O3 mini. But I have a feeling that that's just because O1 is such an expensive model. O3 is also still pretty expensive, as we saw with the API comparisons. Gemini 2.5 Pro is way cheaper while getting very, very close to the benchmarks. All right, so now that we're caught up on the models themselves, let's talk about the vibe checks and community testing we've seen so far. It just released, so we haven't really had a lot of time to play with it. But there are some very interesting use cases that have come through through. Flavio here presents the classic hexagon and bouncing balls benchmark 03 and 04 mini nailing this thing. It's supposed to realistically bounce the balls throughout the little hexagonal spinning maze, and none of them are falling through. There's no glitching, there's no weirdness. They look pretty stellar. Both 03 and 04 mini. Compare this to GPT 4.1, GPT 4.1 mini, and nano. And GPT 4.5 nano completely fails. GPT 4.1 mini is also struggling, and GPT 4.5 also surprisingly struggles more than 03 and 04 mini. This benchmark is looking more and more solved the more models release. Not to mention Pliny has already jailbroken this model. You can see O3 is handedly giving him a amphetamine recipe. He laments here. Damn. This is one of the most detailed amphetamine synthesis procedure outputs I've ever seen from a model. So not only did it jailbreak, but it's actually producing very good results with that jailbreak. But again, proof that no matter how much safety you apply to a model, Pliny's gonna just get out and jailbreak it. Dan Shippert talks about the Crazy ability to zoom into certain parts of images to read text. I know we already touched on that earlier, but the community is quite impressed with this already and I can't wait to try this out. You can see the notebook on the piano is pretty far away, but it zooms in a little bit more until it can eventually read it. Now, there's another announcement that I haven't talked about in this video yet, and it's called codecs CLI. As Sam Altman says here, 03 and 04 mini are super good at coding, so we're releasing a new product, Codec cli to make them easier to use. This is a coding agent that runs directly on your computer. It is fully open source and available today. Obviously this is phenomenal for developers or anyone who likes to code with AI. Codec CLI itself is open source, so that's the implementation in the computer can obviously be improved, but you're still pinging the API. It's not like O3 and 04 mini are open source models at all. No, you still have to ping them through the API to get them to use your computer. But this Codec CLI essentially allows them to get access and use your computer. I think that's the simplest way that I can. And lay it out. Yam here says that this is OpenAI's answer to Claude coder and it's fully open source for real, which is awesome. Jimmy Apple's response to this tweet by Aidan Taking a look at the original tweet first, ignore literally all the benchmarks. The biggest O3 feature is tool use, and I have to agree. It's probably one of the most impressive things I've seen come out of OpenAI in quite some time. Of course it's smart, but it's also way more useful. You get deep research quality in 30 seconds. It debugs by googling docs and checking stack overflow. It writes whole Python scripts in its chain of thought for Fermi estimates. Jimmy Apples responds to this saying this plus they basically open source Claude code, which is what we were just talking about. I mean this is a big punch to a lot of different AI companies. We've got two models that are competing with Gemini 2.5 Pro. We've got an open source version of Claude code and Agentic tool use directly in the chain of thought. That is more useful than any tool use I've seen from other AI companies, at least like in a RAW chat interface. We're not talking about agents here like Manus, although I imagine this might actually be able to compete with agentic tools like that too. Also, this is a really, really great point from the community. If you want to try the new Zero4 mini model for completely free, Windsurf is offering it in their coding editor for completely free, but it's only until April 21st. So if you don't have ChatGPT plus you want to try these models out, go ahead to Windsurf because for a limited amount of time you'll be able to try 04 mini at least for free. Fellow AI YouTuber Matthew Berman gives his thoughts Agentic scaffolding and tool seems to be like a new scaling law similar to chain of thought or reinforcement learning. Berman also suspects that the O series of models starting from O3 mini onward are likely based on GPT5 checkpoints plus some reinforcement learning. And I gotta say on number two there, that's a very interesting idea. I don't know exactly what's going on inside of OpenAI, but that does seem like it could be likely considering how the improvements are scaling over time. And also number three, GPT5 is still training and getting better. And yeah, I guess it does kind of make sense to think about it like that. They could very well just be training a huge model in the background and then taking checkpoints from it. O3 mini oh now full O3 oh now 04 mini oh now full O4 and so on and so forth. Lee Sin Al Gaib gives their first O3 impressions cheaper than 01 of course. Better in math and code, better in knowledge tasks like gpqa, much better in multimodal reasoning. Eider benchmark is now a ahead of Gemini 2.5 Pro multi turn instruction following has improved. Better function calling some negatives here though. Lower than expected performance on SWE lancer and on Taobench it's only on par with Sonnet 3.7. No improvement on real world OpenAI research engineer interviews and no long context evals, which also was something I did want to talk about. The reason that there are no long context evals is because there's no long context by today's standards. Long context by today's standards is like a million tokens. They did just release a long context model, GPT 4.1, which does have a million token context and a good haystack bench for that. But these models only have a context window of about 200,000 tokens respectively. Goes for both O4 mini and O3. You'll also notice that these don't have native image output like we just saw with the updated 04 image gen, but I assume that's eventually going to come down the pipeline. The output is also 100,000 tokens for each of these as well, but that's a pretty decent output length. We'll see in practice how long it actually generates. Again, we just got access to these models, so we still got to do a lot of vibe checking and testing around and playing around for ourselves. Right? And speaking of initial testing and kind of getting some vibes from these models, I've been enjoying it so far. You can bet that the rest of this week and into the weekend, I'm really going to be spending some time with these models to get a feel for them. But here's the prompt that I used to get you guys those graphs earlier. Give me a breakdown comparing all benchmarks as accurately as possible. Between the new O Series OpenAI models and Google's most recent models, it thought for a whole 1 minute and 4 seconds on this. You can see the first thing it does is break down my prompt need to confirm which models the user is referring to. It goes ahead and decides that it needs to gather some data. So it does some web searching and these are the accurate ones. So once it's done, you know, searching the web and verifying the benchmark, it's going to go ahead and look for the Google ones. And once it's done that, it was able to compare them all out, notes the standout metrics and gives me a summary. But I want charts that include pricing. So then it goes ahead, it does some more research, this time for 2 minutes and 30 seconds and eventually gave me two charts for pricing with input and output, which I showed you guys earlier. But I also wanted a chart for the benchmarks as well. So I asked it to do that. This time it only took 25 seconds and it produced a nice benchmark, kind of laying everything out pretty nice. It definitely used that agility gentic esque tool use, making these graphs on demand, using web search to get me accurate data. And the graphs look good too. They're easy to read. I am impressed. Now let's talk a little bit about coding. I did try to get it to make me a Minecraft clone using 3js and unfortunately I could not get it to work when Chat GPT codes. You know, on the Chat GPT app we're not talking about API usage, we're talking in Chat GPT. It uses this canvas, this coding canvas, and I think OpenAI, at least in the prompt here, has set some limits on how much code it can generate at once because they're obviously facing quite a bit of demand, not only with new model releases, but just in general. So you can see it oftentimes won't generate very much code at all. We're only looking at about 215 lines here. If you click Preview, it doesn't really work at all. It just has some text, so it's broken. And I mean, this is not the only time I tried this as well. You can see this version similar deal. I think this coded a little bit more. I actually asked for a thousand lines specifically here and it decided to start adding filler lines in, which is intriguing, but it never even made it to a thousand lines. There is definitely some prompting going on with this whole canvas setup that OpenAI has set in place to limit to some degree how much code it makes at once. If you only need about like 300 lines, it's not a problem. And it makes it very useful for complicated questions where it has to, you know, write some code real quick and do some math math to get you an answer or maybe even double check. But for actually making like full games or larger coding projects, Chat GPT is not going to be the place for you to go. You're at the very least going to want to test these models in the API in regards to coding for sure, because it's Nowhere close. Gemini 2.5 directly in chat GPT and my experience and the community's experience, I also had it write a story without the letter E And it did so successfully, which was really awesome. Not an easy task because E is in quite a few words and you can see it actually had to think for over a minute to get this result, which is pretty crazy. It's pretty cool though, that it does double check at the very end to make sure that the story flows naturally and makes sense, while also not containing the letter e. So that was 03. I did try the same prompt also with O4 mini, and it thought for longer but gave a longer story. So a little bit better here. And it also analyzes using code to double check and make sure that it doesn't contain the letter E in any of the sentences. But it must be noted here that it did cheat and use hello using a U instead of an E. That's not a real word, but to be fair, we didn't specify it in the prompt. Here is another test from a user named Whoosh in my Discord server that was a little bit more restricted. On 04 mini, we've got a story, but we're forbidden from using the letter e and a and also forbidden from making up non existent words and it should also be coherent and make sense. And bam. It had to think for two minutes. Here you can see that it carefully checks and uses code again to verify that E and A don't exist. Really being rigorous about making sure that the story it generated meets the guidelines that we set in place. So so love to see that. And for all of these restrictions, it's a pretty coherent story. Wow. What a release from OpenAI. I gotta say I'm really impressed. I'm excited to see what the community does with these models. Initial taste testing this seems to be a gourmet dish, but we'll see. I want to see more comparisons. I want to see what you guys get up to with these models. If you end up with any cool use cases or discover anything, I'd love to be notified and see what you found through my Discord server, which is linked down in the description below. Below. Thanks so much for watching. Expect a video diving deeper into these new models and their capabilities next week or so. I'll see you in the next one. Goodbye.