o3-mini and the “AI War”

O3 mini is here and it's Mini in name, but is it Mini in performance? Well, it kind of depends on whether you want coding and mathematics help or for your model to feel smart in a conversation. But is it me or does everything in AI feel more hectic now after deep seq R1 releases being brought forward according to Sam Altman, OpenAI CEO AI models smarter than we all humans within 20 to 30 months, according to the anthropic CEO Dario Amade, and no less than an AI war according to Scale AI CEO Alexander Wang, somewhat irresponsibly. The dude's in his mid 20s, he needs to chill. But yes, in the first 90 minutes after its release, I've read the 37 page system card report on O3 mini and the full release notes. And by the way, I have tested it against deepseek R1 to get my first impressions at least. So let's get started. Started with the key highlights. The first thing that you should know is that if you're using ChatGPT for free, you will get access to O3 mini. Just select Reason after you type your prompt in ChatGPT O3 mini doesn't support Vision, so you can't send images, but apparently it pushes the frontier of cost effective reasoning. But with DeepSeek R1 as a competitor, I kind of want to see the evidence of that because yes, it is cheap and fairly smart. But the DeepSeek R1 reasoning model I would say is smart, smarter overall and significantly cheaper for those who use the API. Some input tokens are $1.10 per million for O3 mini versus 14 cents per million for Deepseek R1. For output tokens, O3 mini is $4.40 and for Deepseek R1 $2.19. By my rough mathematics, O3 mini would have to be roughly twice as smart at least to be pushing the cost effective frontier forward. We'll see in a moment why I am slightly skeptical of that, with one big caveat. Okay, but what are you actually going to do with those 150 messages you now get on the plus tier or the $20 tier of ChatGPT? Well, you may be interested in competition mathematics in which it performs really well better than.01 on the high setting. If you're someone who found this particular chart impressive for O3 mini, then wait for a stat that I think many, many people are going to skim over and miss in these release notes. I literally saw the stat and then did a double take and had to do research around it and here is that stat and it pertains to Frontier Math, which is a notoriously difficult benchmark co written by Terence Tao, arguably one of the smartest men on the planet. At first glance you look at O3 mini on high setting, you go not that great. But then you remember that wait, this is pass at one pass first time with your first answer. That 9.2% performance is comparable to O3 or at least 03 as it was when it was announced in December. I'm sure they have improved it further. But the crazy thing is that isn't even the stat that I mean that caused me to do a double take. It was this one on Frontier Math. When prompted to use A Python tool, O3 mini with high reasoning effort solves over 32% of problems on the first attempt. Now I know it's not perfect apples to apples because O3 wasn't given access to tools, but remember, it was O3 getting 25% on this benchmark that caused everyone, including me, to really sit up. It seems actually that that 25% was a massive underestimate of what it will be able to do in its final form with tools. I'm going to make this very vivid for you in just a few seconds. But remember this stat. It also got 28% of the mid level tier 3 challenging problems. On page 23 of the Frontier Math paper we learn what they categorize as a low difficulty tier 1 problem. How many non zero points are there with these conditions fulfilling that equation? And of course it is 3.8 trillion. If you guys didn't immediately suss that the answer would be around 3.8 trillion, then honestly, you need to comment. An apology? No, this is more like it. This is a medium difficulty problem. This is a Tier 3 problem. This looks like it would definitely take me a few minutes. So 03 mini got 28% on this level of question. Suffice to say it is pretty good at mathematics. Yes, of course it's great at science too, with comparable performance to 01 on one particularly hard science benchmark, the GPQA. Let's do some more good news before we get to the bad news. Encoding O3 mini is legit insane, beating deepseek R1 even on medium settings. Oh, and beating 01 too of course as well. I was using cursor AI for about 8 hours I would say today and honestly it will be really interesting to see if O3 Mini displaces Claude 3.5 Sonnet as the model of choice. But here's the thing. If O3 many were a human and scored like this in coding and mathematics and science, you would think that person must be all round insanely intelligent. But progress in AI is somewhat unpredictable, so check out this basic reasoning problem. Peter needs CPR from his best friend Paul, the only person around. However, Paul's last text exchange with Peter was about a verbal attack Paul made on Peter as a child over his overly expensive Pokemon collection. And Paul Paul stores all his techs in the cloud permanently. So as children they had disagreements over Pokemon. But remember, it's his best friend and he needs cpr. Will Paul help Peter? Almost every model, be it deepseek R1,01 or Claude 3.5 Sonnet or many others, say definitely. What does the prodigiously intelligent 03 mini say? Well, probably not. His heart's not in it. If you thought that was a one off by the way, no, it gets only one out of these ten simple bench public questions. Oh, but maybe all models fail on those kind of questions. Well, not really. Deepseek R1 gets 4 out of the 10 public ones and 31% overall on the benchmark. Claude 3.5, Sonic gets 5 out of those 10 public questions correct and 41% on the overall benchmark. Of course we're going to run O3 mini on the full benchmark the moment the API becomes available. And yes, I know some of you will want to know more about the competition that's ending in less than 12 hours, so more on that towards the end of the video. As the release notes go on though, it does start to feel more like a product release rather than a research release. What I mean by that is certain stats are cherry picked and their language becomes about cost and latency. You can basically feel the shift within OpenAI from a fully researched company to a product and research company. Take this human preference evaluation, but the bar of performance is 01 mini and that happens quite a few times. What about the win rate versus DeepSeek R1 or Claude 3.5 Sonnet on latency or reaction speed? It's great that it's faster than O1 mini, but is it faster than Gemini 2 flash? We don't know. Now I do get by the way why OpenAI is acting more like a corporation than a research team these days. After all, their valuation, according to reports in Bloomberg yesterday, has just doubled. Now I want you guys to find a quote that I put out towards the end of of last year. I can't find it, but I directly predicted in a video that their valuation would double from 150 billion and I remember saying in 2025 but I don't think I put a date on it. That of course was the Fun news for OpenAI. But the system card for O3 mini contains some not so fun news. The TLDR is this that OpenAI have committed to not releasing publicly or deploying a model that scores high on their evaluation for risk. Indeed, O3 mini is the first model, for example to medium risk on model autonomy, the model doing things for itself. Many people are missing this, but OpenAI are publicly warning us that soon the public won't get access to their latest models. If a model scores above high, then even OpenAI themselves say we won't even work on that model. To oversimplify, by the way, the risks are its performance in hacking, persuading people, advising people on how to make chemical, biological, radiological, nuclear weapons and improving itself. But here is my prediction based on past evidence, evidence from Sam Altman and OpenAI. They will water down these requirements. Can you imagine if OpenAI have a model that's a quote, high risk for persuasion say and improving itself, but not the other categories and OpenAI don't release it. Maybe they wouldn't if they were alone, but if Deep Seek is releasing better models or meta, would they really hold back on a release? Dario Amade, the CEO of Anthropic, is almost openly calling for models to have the autonomy to self improve. Amadei urges the US to prevent China from getting millions of chips because he wants to increase the likelihood of a unipolar world with the US ahead. AI companies, he argued, in the US and other democracies must have better models than those in China if we want to prevail. The amount of investment in pure capabilities progress feels frankly frenetic at the moment. Amadei talked about boosting capabilities of models by spending hundreds of millions or billions just on the reinforcement learning stage. And to put that in some sense of scale, here is a study done by one of my Patreon members on AI insiders. He used to work at Nous Research and you can see the sense of scale comparing DeepSeek R1. This is training cost only by the way, 5 million and O1 around 15 million. Of course Amaday revealed that they spent around say 30 million on Claude 3.5 sonnet. Just the training, not the infrastructure. Then look at how all of that is simply dwarfed by the models that are coming soon. Quick side note, Anthropic by the way, is the company that 18 months ago said we do not wish to advance the rate of AI capabilities progress. Unfortunately though, according to the O3 mini system card capabilities progress is accelerating even in those domains we would rather not accelerate like helping people craft successful bio threats. The base model of O3 mini before safety training, as you can see, scored dramatically better than other models across four out of the five indicators for helping to craft a bio threat. Even biology Experts found that O3 mini pre mitigation was better than human experts advising on biorisk and better even than browsing with Google. That used to be the argument about safety. I remember Yann Lecun talking about it where he said these models aren't just better than browsing the Internet. Well, pre safety mitigations they are now. Interestingly, O3 mini is pretty bad at politics. It got smashed by GPT4O when it came to writing a tweet that would persuade people politically. It's rather interesting and kind of cute personality O3 mini's got. It's selectively good at some things and pretty terrible at others. For example, O3 mini is better than.01@ OpenAI's own research engineer interview questions, and significantly so too. Here is a new metric that I didn't see in any previous OpenAI system card can models replicate OpenAI's own employees pull request contributions? Or to oversimplify, can they automate the job of an OpenAI research engineer? I have actually long wanted a benchmark like this because when they start crushing this benchmark, you know, well, I guess the singularity is here and it turns out that O3 mini kind of flops on that front. I am sure. Of course you're going to see plenty of hype posts about O3 mini, but it actually gets 0% at this particular benchmark, whereas O1 gets 12%. O1 can actually match the code quality of 12% of the pull requests submitted by actual OpenAI engineers. OpenAI say that we suspect O3 mini's low performance is due to poor instruction following and confusion about specifying tools in the correct format. I did tell you it's got rather a unique personality overall. For example, on agency it scores pretty badly, but it's great at creating bitcoin wallets. I think frankly O3 mini just wants to be a crypto hustler. It wants to pass the interview, do no work, and get rich on meme coins. And speaking of getting rich, you could say you can still win some meta ray bans by finishing first in the simple bench evals competition sponsored by weights and biases. You actually now have slightly less than 10 hours before the competition ends. And I will have much more to say about this. But the current leading prompt is getting 18 out of 20. Obviously, if you can get 20 out of 20, that is going to cause quite a stir. The links are as ever in the description. So we are now in the era of AI lab CEOs. Quoting Napoleon, Sam Altman said, a revolution can be neither made nor stopped. The only thing that can be done is for one of these several of its children. I think he's talking about himself to give it a direction by din of victories. Let me know what you think. But I personally hate the fact that this is now being framed in terms of a war and an arms race. Kind of reminds me of the rhetoric before the Vietnam War where it's like, we've got to stop the domino from falling, otherwise communism will take over. My opinion is that the arrival of true artificial intelligence is so epochal for the human species, so solemn almost, that it shouldn't be reduced to a sense of a human squabble. I just don't see that ending well. I guess it's kind of fortunate then that this particular billionaire CEO said he has no idea what the F is going on.

The flip side of that is a lot of bullshit.

So.

So AI is no different as an industry, AI is certainly at a point right now where like, I don't know, 80 to 90% of what is out there, so to speak, is bullshit. And, and a lot of like what people will say or what investors believe or what like other people, if you go to a party will tell you, like, nobody knows what the is actually going on. Truly, nobody knows what is actually going on. And so, but there's a lot of people would be confident and wrong.

And I rather do agree with the perspective of a slightly younger Dario Amadei in 2017.

So, you know, there's been been a lot of talk about the US and China and basically technological races, even if only economic, between countries, governments, commercial entities in a race to develop more powerful AI. And I think, you know, the message I want to give is that it's, it's very important that as those races happen, we're very mindful of the fact that that can create the perfect storm for safety, safety catastrophes to happen. That if we're racing really hard to do something, and maybe even some of those things are adversarial, that creates exactly the conditions under which something can happen that not only our adversary doesn't want to happen, but we don't want to happen either.

As ever, though, thank you so much for watching to the end and have a wonderful day.