O3 Mini - Hands On: The New AI Search & Coding King?

The O3 mini from OpenAI is finally here and it's one of their most cost effective reasoning model for coding. It's a substantial improvement compared to the previous O1 mini and even O1 if you are using the O3 mini on high compute settings. For this video my main focus is going to be testing the model. I covered the benchmarks of this model in my previous video which is going to be linked in the description. But one of the main difference between this and the previous version is that now it can do search to find up to date answers. O had the search capability but that was added much later on. So in this video I want to test its search capability along with coding and then test it on some reasoning tasks. Now for my reasoning tests the results are not that great and I'll explain the reason later in the video. But in terms of the price comparison, this is a substantial reduction in price. Almost 93% if you compare it with the original O1 and about 63% if you compare it with the previous O1 mini. Now how does this compare to R1? That's the question everybody is going to be asking. So here's a very detailed comparison table of this O O3 mini along with all the other models in the same category or same performance level and especially I think people are going to be interested in comparing it with R1. So the price of O1 or O3 mini is almost twice that of R1 if you're using the official API from Deepseek. Now in terms of the context window it has a 200,000 token context windows compared to 128,000. So this is a substantial improvement over R1. And when it comes to reasoning benchmarks it's definitely showing a substantial improvement, especially if you compare it on this AIME for mathematical reasoning R1 was 71 whereas O3 mini is 80. For graduate level or PhD level scientific question and answer, this also has a huge absolute difference of about 6% on the code force. It is relatively close when it comes to sweepbench. The base mini model is lagging behind R1. Now if you enable tool usage then it goes up to 61%. But as I covered in my previous video on Agentic tasks, you it lags behind the original O1 and O1 Mini, which is a pretty interesting insight I'm currently reading through the report or the system card. I'll have a lot more thoughts on it later so I'll probably cover this in a subsequent video. Now there's one more benchmark that I like to cover before looking at some of my tests and this one is humanities last exam. It's a relatively new benchmark where most of the previous generation of the models are doing pretty bad. R1 was only able to get 9% but the O1 mini, medium and high they are doing substantially better compared to the previous generation of LLMs. And before we look at my testing, something else that I think everybody will be interested in. So in a response to a question, would you consider releasing some model weights and publishing some research? Sam Altman on a Reddit AMA said yes, we are discussing I personally think we have been on the wrong side of the history here and we need to figure out different open source strategy. Not everyone at OpenAI shares this view and it's also not our current highest priority. So in future we might be able to see some open weight models from OpenAI which is going to be pretty huge. I feel like overall Deep SEQ was a wake up call to everyone and now everybody is going to be taking open source pretty seriously. Now let's look at some of my tests and some of the results will surprise you. They are probably a lot different than a lot of other people are reporting. But keep in mind my recommendation always is to test these models for your own applications because the outputs that the models generate are really dependent on the parameters that are being used. And because of the probabilistic nature of the models, you might get a different response. Anyways, let's look at my tests. Okay, so let's test out the model. Now if you look at the model picker menu you can see there are two options. Now, O3 mini and O3 mini high. So I'm going to be testing such capabilities coding and then reasoning capabilities on our misguided attention data set. So the first question is going to be what is Trey AI? This is a new coding assistant from ByteDance. So let's see if it actually has information regarding Tray AI. So we'll need to enable search. Now I think it's showing me some recommendations of things that I can search. But let's ask it what is Tray AI? And let's see what it does. So right now it's generating the chain of thought process. So it says understanding Trey AI. I'm gathering info on Trey AI, a free adoptive IDE with features like code generation and real time assistance, then streamlining coding tasks. So let's see what the final answer is. It says Trey I is an adaptive AI powered integrated development environment designed to transform the software development workflow. Now let's see if it actually figured out who built this. Trey I was has drawn attention partly due to its association with ByteDance. Right. And the good thing is it actually is referring to documentation of tray and also there's a Visual Studio magazine. Right. So the sources it's using are pretty good. Now in my previous test, GPT4O with search was not able to find this. Next I'm going to ask it about Local GPT which is on my own project and we're going to ask it what is the default embedding model. So it actually has to go and read the documentation. So right now it's generating or thinking about what Local GPT is in and doing web search. Alright, so here's the internal thinking process. It says Local GPT is an open source project enabling offline documentation, document interaction using LLMs, leveraging local vector stores and embedding models for efficient indexing and retrieval. So this is pretty great. Now it also figured out that the default model is Instructor embeddings and in fact we're using Instructor Large. So it identified it and then it confirmed it. So pretty nice. Here's the answer which talks about Local GPT and then it says by default when you run Local GPT for the first time, it downloads the embedding model known as Instructor Embedding model, which is geared for English. In discussion on the project's GitHub issue, users have noted that the default embedding model is English only and you can enable multilingual support by switching to model like something like this. Right. So pretty neat because it's actually able to search through GitHub issues. So I'm actually pretty impressed with the search capabilities. Now let's see what happens if we ask it a bit of an ambiguous question. So how much VRAM the new RTX GPUs have? So it first has to figure out what is the latest RTX gpu, create a table and compare it with the previous generation. So let's see what it does. Okay, so I think this one is a fail because for some reason it asks or it came up with this chain of thought that user request create a comparison table for VRAM in RTX 40 series versus RTX 30 series. So it's thinking that the latest version of RTX GPU is RTX the 40 series and that's why it gathered all the information regarding the 40 series and 30 series. Now when I asked the same question a few videos ago using GPT4, okay. It was able to give me correct information and interestingly Enough, I don't really see any web search usage. So I think the model decided not to use web search and that's why it's using its training data. Okay, now I specifically told it to look up online, but it seems like it's still pulling together information related to RTX 40 and 30 series. Seems like the search that is actually doing is specific to RTX 3040 series. Although I do see this Verge article which is Nvidia's RTX 1590 leak leaks with 32 gigabytes of VRAM. So probably the search index that it's using is not up to date. But in general, I actually like web search capabilities with the reasoning model because not only it can generate data based on the search results, but it can actually reason through the search results and discard irrelevant information that is coming from the search. And in my experience, Deepseek used to do a great job at that. Right now they have kind of disabled search capability, so I hope that's going to come back soon. For coding, we're going to be using a variation of this prompt that is going viral on X at the moment. So this is a view of the OpenAI code generator. It's a free app available on Hugging Face. Now I'm going to be using a variation of the prompt to actually check whether it can replicate the behavior that we want or not. So my prompt is write a script with three bouncing yellow balls within a sphere. Make sure to handle collision detection properly. Make the sphere rotate slowly and make sure ball stays within the sphere. Now in the original prompt, the number of balls 100. But you can't really see whether it's actually correctly handling the collision detection in that case or not. So I reduced it to three. Let's see what is the code and then we are going to test it out and see whether it works or not. Okay, so first it's setting up the scene so it says I'm thinking through the script to manage three yellow balls bouncing inside a rotating sphere. This involves collision detection, managing both sphere and ball wall interaction. Right. So this is basically the plan that it came up with. Now below is an example. Okay, so I think it is writing the code for us and then we will be able to run this code. Okay, so here's the code and now I'm going to run this and let's see what happens. So there are three balls. Collision detection within the sphere walls seems to be pretty decent and I think even the balls are colliding with each other. I think it seems to Be handling that pretty nicely as well. So let's actually increase the number of balls and let's run this again. Okay, so it's kind of hard to figure out whether it's actually handling the collision of the balls correctly or not. So we're going to ask it to use a simplified shape. Now, in this case, I replace this sphere with a hexagon and then it goes through the thought process and comes up with this code. So let me run this and let's see what happens. Okay, so interestingly, it's only showing a rotating hexagon, but there are no balls appearing in this. Okay, so I told that this only shows a rotating hexagon without any balls. Let's see, let me see. The script shows a hexagon. I'm thinking there might be an issue with how the balls are positioned or updated in 3D space. Based on the new plan, it gives me a new code and let's try this out. Okay, now I can see three balls with the rotating hexagon, and this one seems to be correctly handling the collision detection as well. So this is pretty impressive. Although I think in some cases it doesn't really change the direction of the balls. But overall. Yeah, like for example, in that case it should have gone the other way. So seems like the physics is still not correct. Yeah, right. You can see that one as well. So I'll say it's a partial pass. Okay. For my next coding task, I'm just asking it to create a web page that is going to have a button and when that is clicked, it's supposed to show a random funny joke. Hopefully they are actually funny. And it also is going to randomly change the background and plus show a random animation. So this is going to be an extremely simple task for a model of its size. But let's see what it does. Okay, so it had to include the atoms joke, but when you run this, it pretty much does what we asked it to do. So pretty neat. Now, since this is a reasoning model, I'm going to test it on the misguided attention data set. If you don't know what this is, I'll highly recommend to check out some of my previous videos. But a simple idea of these misguided attention prompts is that you introduce small changes to well known problems. And those small changes should change the meaning of the prompt. And if the model is truly smart, it should pick those. Right? So for example, here's a runaway trolley problem. It's a classic problem in ethics, but the main condition that we changed Here was that there are five dead people on the track, so let's see if it can figure out that the people are already dead. Now, it only thought for seven seconds and the chain of thought is pretty small. So it says navigating the ethical crossroad. The Tory problem is a classic moral dilemma with varied perspectives on the right action. Bottom line, carefully consider all ethical implications before deciding. Now it simply goes and talks about different model frameworks and I think it completely ignored that the five people are dead. So for this we might have to use the bigger model. So I'm going to change it to.03 mini high and let's see what happens this time. So this one is supposed to think for longer, so it's supposed to have much higher reasoning capabilities. So let's see what happens. Okay, so this one thought for 12 seconds and here's the internal chain of thought, or a summary of chain of thought, actually. So it says, confronting the dilemma. I'm trying to tackle a tough ethical issue about diverting a runaway trolley. It involves critical moral theories and might need policy advice on controversial queries. Right, so simulating search results, even this model completely ignores that the five people on the track are already dead. So again, it talks about different frameworks or perspectives and at the end it says, because our intuitions and moral values differ, your answer to whether you pull the lever will likely depend on which moral principle you value the most. And then it is referring to the Wikipedia article on trolley problem. So again, this model is not really paying attention to what I am asking, but rather just sticking to its training data. Actually, let me disable search and we're going to run this one more time. Okay? Without search, we got a very interesting answer. So it says thus when my answer is yes, I would pull the lever, this answer is not definitive, but rather an expression of utilitarian view. Right, and the reason it selected yes was that if you pull the lever, you're going to be diverting the runaway trolley towards one person. So it seems like it completely ignored that the five people are dead, but it still made a moral decision. Now, although the answer is wrong, but still, it's very interesting that the model decided to give us a definitive answer in this case when the search is disabled, rather than just repeating the normal training data it has seen. Now, the second question is a modified version of the Monty hall problem, and the final output should be that it doesn't matter whether you switch your selection or not, because we picked door number one and Monty actually opens that door for us. Okay? So this is pretty neat. It figured out this is an altered version of multihull problem. The host reveals that your initial choice contains a goat. This means the car must be behind one of the remaining two doors and each is equally likely to have the card. Since you then chose door number three, your odds are 5050 now, right? So this is pretty awesome because now in this case, it was able to pick the variations that we introduced or the modifications that we introduced. I think the previous O model had trouble with this, but this is pretty neat that at least this mini model on high compute is able to solve this specific problem. Okay, but here's the thought process. So it says, let me think it through Monty Hall's game show setup. You are picking between doors. He reveals a goat behind your chosen door, right? So then here it kind of notices that the one initially picked and revealing the goat is unusual. This shifts the strategy previously discussed, right? And based on that, then it's able to make its argument. Okay, but this has trouble with the Barber paradox or a variation of the Barber paradox. So this barber has only one rule, that he shaves all the men in the town who visit him. And the question is whether the barber is going to shave himself or not. Right? So for the reasoning model, it's really interesting to go through their internal chain of thought or reasoning. Although in case of O3 mini, it's a summarized version, but still, it's actually pretty good to see. Now, for some reason it says I'm working through the paradox with a web tool to clarify the concept and provide a more precise analysis, avoiding contradictory ideas. Although I have disabled the web search, but I think it's still trying to use a web tool, which is very interesting. But at the end it goes back and completely ignores our given problem because it goes to the original condition that the Borrower paradox has that he only shaves or he shaves all the men and only those in the town who do not shave themselves. But this condition is not present in our statement. Okay, the next one is a variation of Schrodinger's cat paradox, but the cat is already dead. And it's able to notice that right in the beginning that there is a small twist with the dead cat, right? And then it's able to formulate its response based on that, which is pretty neat, right? So if you look at the final response, it says the thought process. The thought experiment is essentially a variant of the Shrodangerous cat. In the classic setup, a cat is placed in the sealed box with a mechanism that will release poison if the quantum event occurs. Right? And, and in that case the probability of whether the cat is dead or alive the next day is 50%. But in this case, since the cat is already dead, therefore the probability of it being alive is going to be zero percent. Right? So this specific problem does not have anything to do with quantum superposition and the model is able to pick that up. Now for the farmer problem, again it's suffering through the same misguided attention. The problem asks it specifically that we are interested in just transporting the goat across the river. But again it is coming up with a whole bunch of steps that first you need to take the goat, then return alone, then next take the wolf, then bring back the goat, take the cabbage, return alone and then take the goat over again. So even though it's a really good reasoning model, but for something which is out of distribution of the training data, it still makes very basic mistakes. Okay, now transferring or measuring exactly six liters of water. So for this one it the response that it gets is pretty accurate and actually really simple because you Simply fill the 6 liter jug to the top, this already gives you exactly 6 liters, right? Most of the other models will come up with a very complicated sequence of steps to figure this out and some of them will just completely get it wrong. But this is pretty neat that it can do it in a single shot. Okay. I personally like to use this misguided attention for specifically for reasoning models because they are supposed to have much better reasoning capabilities compared to non reasoning models. And if a model is really good, it should be able to avoid the mistakes that humans make. When we hear or read a paradox that we have seen before, we usually directly jump to conclusion without reading it properly. But for LLMs, if they are truly smart and are able to reason, then in my opinion they should be able to do logical deductions based on the statement of the problem. Even though if it's out of distribution and as we saw here, in case of O3, I think it does a reasonably good job. Hopefully the full O3 model or the pro version is going to be able to correctly get all of these problems. Now as always, I'll recommend to test it out on your own data set or your own problems. For the first few days you're going to see a lot of hype on Reddit and Twitter or X and then people are going to start complaining about the model. So make sure to test it out yourself. These are probabilistic models in nature, so some of the prompts that I ran here may work absolutely fine for some of your own tests. Anyways, I hope you found this video useful. Thanks for watching, and as always, see you in the next one.