AI NEWS DROP! Google Strikes Back, o3 & o4-mini tests, Open Source AI Video!

This might have been one of the largest weeks in AI. We'll see all of 2025. OpenAI dropped two big new models that are really making the rounds and now the community has had some time to see what's what regarding them. So we're going to be talking about that today. But that's not all. We're also going to be talking about new video generation enhancements, several new models in that domain, and not to mention Google did a little bit of a strike back on OpenAI and Grok might also be seeing some updates. Let's check them out for the 03 and 04 miniseries of models from OpenAI. The community has had their time to taste and kind of play around with the models. Talen here notes that the most impressive thing they noticed about O3 is that it finally stopped answering everything with a list and now it answers everything with a table. And I gotta say I've also noticed that in my own testing I've been having this thing write all kinds of different research papers. And while I have to say I've noticed that O3 still is pretty short in terms of the actual word count and the amount of of depth in comparison to deep research, the quality of the generations, the quality of the actual text and research is very much high. The creative ideation and the intelligence really does shine through and you'll notice. Yup, we also see tables, data stream versus Species versus Metric versus prediction. These can be very potent and offer a lot of information at a quick glance all at once. Now in this case you're going to have to know some things about like biology. I think in order to understand this at a quick glance, this was a little bit more of a simple prompt getting a full length report on a specific dog breed behavior. I gotta say again, quite short, lots shorter than you typically would see with deep research. But the tables are quite nice, the information is laid out well and it's giving us the key points that we need at a glance. It's not going too overboard and I think that honestly might be preferable for a lot of people. At the end of the day, not everyone has time to sit down and read a 10 minute long research paper. This is probably a quick 5 minute read and it gives you nice clear tables with all that information right at a glance. Even the original GPT4 would sometimes give you tables, but not this frequently. They are clearly a very efficient way of conveying information and ideas through text and I think Grok3's release originally made everyone realize it and now OpenAI also knows it's a great idea, so they have made sure that these new O Series models provide plenty of tables. Now, speaking of leveraging tools to make life easier, if you're a freelancer, a side hustler, or a consultant, managing the business side of things can get chaotic fast. But that's why I want to give a quick shout out to today's sponsor, Honeybook. Honeybook is essentially the leading client CRM platform designed specifically for entrepreneurs and independent business owners. If you're juggling clients, projects and trying to actually get paid for your awesome AI work, you know how messy it can really get. Thing is, they've actually been doing this for a decade, helping folks manage everything from leads and clients to project details and finances. It can all be tailored to how you at the end of the day and help give your clients a really personalized, polished experience. Honestly, the biggest thing for me, and probably for many of you watching too, is getting bogged down in admin tasks with Honeybook. So much of that manual stuff like sending proposals, contracts, invoices, scheduling meetings, it just gets all automated. That means you can actually spend more time focusing on the cool stuff like diving deep into AI projects, creating content like me, or working directly with your clients. And yeah, you already know it. HoneyBook has AI features as well, almost letting it become like a behind the scenes partner. Think drafting client emails faster, getting meeting notes summarized automatically, or creating project recaps all powered by AI, freeing up even more of your brain power for the creative or technical work you actually enjoy. As I've grown my own little business around content creation and AI, at first I thought it would mostly be about the tech and while that is true, suddenly there's all this extra business stuff to hand. Honeybook is like getting a business partner to help me streamline all that admin stuff, giving me valuable time back to actually focus on what I like doing the most. So if you are an entrepreneur or small business owner like me and you're tired of juggling spreadsheets emails, definitely check out Honeybook. It's completely free to try, no credit card required, which is great to see. All you got to do is hit the special link I've put at the top of the video description. So give it a go. See how it streamlines your workflow. Huge thanks to Honeybook for sponsoring this part of today's video back to your regularly scheduled content. Alright, moving on. We've got plenty of new benchmarks tested by the community to check out. First up by Spencer Schiff 03 is State of the art on Enigma eval. This tests for the ability to solve complex multimodal puzzles, meaning vision included. And even though the absolute score is still low, this is still the biggest leap we've seen so far in a specific model. And when he says that the absolute score is still low, it means it got 13% pass rate. That is super low. That's an F minus. But even Claude 3.7 is down here at a 2.26% pass rate. So 03 medium here is dominating in this Enigma benchmark, almost a double leap over the also new dominating model, 04 mini. Even Gemini 2.5 Pro experimental is pretty far down on this list and I think it's down to that multimodal puzzle. Gemini 2.5 Pro is a phenomenal model, but when it comes to multimodal OpenAI just flatt everybody, no questions asked. And there was a huge bump with this latest release of 04 and 03 series of models where they already led in that space. Some quick information from TBOR on the usage limits inside of ChatGPT for the new OpenAI models. Plus, team or enterprise accounts have 50 messages per week with O3 and 150 messages per day with O4 mini. And again, this points towards O3 being pretty expensive for even OpenAI to run at the end of the day, but it definitely shows in certain benchmarks. You also get 50 messages per day with O4 Mini High. And if you're wondering what the difference between 04 Mini and O4 Mini High is, 04 Mini High just does more reasoning. It spends more time thinking. Now, if you pay for the insane $200 a month ChatGPT Pro subscription, you can get unlimited access to all models, but that's to be expected when you're spending that much on an AI plan. Now, if you're also looking for information on context, length, size, TBOR provides that as well. Free plan. Your context window is limited to 8,000 tokens plus limited to 32,000, which is kind of disappointing. I wish this was closer to 50, 80. 100,000 tokens plus is supposed to get, you know, plus benefits. Pro is still limited to 128,000 and Team and Enterprise 32K and 128K respectively. Disappointing. Honestly, I'd like to see higher context window lengths because this is going to lead to people using the models in ChatGPT and thinking they're not great at long context stuff, when really the models are just getting handicapped by OpenAI because they're trying to save costs. Not enough GPUs to go around here and it shows. I will say though, if you use these models in the API you get the full up to 200/token context window, I think. Alright, a couple of examples with these new models. 04 mini high. Surprisingly, this is a difficult question for AI models. They can be so capable at complex and deep tasks, yet falter at even the simplest of math equations. But in this case we can show that it finally is getting some of these questions right that AI models typically used to struggle with 9.11 minus 9.8. Typically I think it would even give you a positive answer instead of a negative one. But this one's given us a negative 0.69.04 mini high seems to do a pretty good job with this. Next up we've got Technicanche asking 0.4mini high to create an autonomous snake game where the snakes actually compete with each other for the apples. And you can see it absolutely did that. The AI is going for those apples as soon as possible. You can see they're kind of fighting each other. A lot of people are talking about how these models aren't great at coding, but it turns out that's not really true. It's only when you're using the models in chat GPT. Again, we have a smaller token context window. We have Pre prompting from OpenAI. If you use these models in the API for coding or codec cli, they're actually pretty darn good. But comparing to Gemini, Gemini still wins out at the end of the day. Next up, we've got 03 doing something kind of creepy. DD gave it an image of a menu of his favorite Chinese place in San Francisco with no title or XIF data. And it was able to search the web, match menu items and locate that restaurant, which is pretty awesome, but yeah, kind of creepy. We just send it a photo of a menu and it can just figure out exactly what restaurant that is. Oh, and don't forget it's gonna give you the breakdown graph of the location and all of its information. Pretty crazy that this works though. I've seen other examples where people simply send like a selfie of them in some location. It's able to pinpoint the location exactly. Again, creepy. Adi Ganesh here provides us with some really nice, detailed insights on how intelligent the O3 and O4 miniseries of models really are under the hood. Adi laments that he loves the way O3 can answer complex queries about research papers. For example, and stay with me here. I don't know exactly what this is but it's a PDF of a cell type Signatures in cell fragmentation profiles reveal disease biology From Nature Communications 2024 paper. Not my field of study, but he asked it to summarize a figure for him and find similar literature and only 41 seconds of thinking. It really impressed him at this point in time. These large language models, at least on these literary tasks, are way past a high schooler and getting into college graduate territory and I bet you could push it farther, you know, into doctorate territory. I think deep research definitely touches into that territory. With the right prompt. These things can be superhuman. And even when you lazy prompt them, you're getting graduate level insights. And yeah, you're getting these insights at speeds that are way faster than human experts. Scott Swingle here points out that O4 Mini High just solved the latest Project Euler problem from 4 days ago in just 2 minutes and 55 seconds. Far faster than any human solver could hope to complete. Only 15 people total were able to solve it in under 30 minutes. So we're talking 10 times faster with AI and it's able to do it correctly. And that's compared to human experts. The fastest human experts are 10 times slower than this brand new 04 mini high model. And yeah, I mean this is the Euler problem. That looks like some pretty complicated math. I was never great at math in school and definitely would not be able to solve this. The AI is smarter than me, but also as smart as human experts while being 10 times faster. Epoch AI has been releasing benchmarks as well on these brand new models. Again, this is them evaluating. This is not OpenAI's evaluations. This is third party. If we take a look at this benchmark, you'll see the highest model is actually 04 mini with high reasoning getting a 15 to almost 20% on frontier math. Shockingly, O3 mini is right behind it, followed by O3 which is the new model. So 03 not doing too great here. Barely better than the original 01. But O4 mini is absolutely dominating. It is important to Note here, Gemini 2.5 Pro has not been evaluated on this benchmark yet. So it might come in first place. Don't exactly know just yet. Tibor ran this very interesting experiment pointing out how difficult it still is for these highly intelligent AI models to solve simple problems like answering what time it is on an analog clock. You can see all he did was upload the photo of the analog clock. It's about 8:52, which it does get the correct answer, but it had to think for over seven minutes. To get that answer, which is crazy. Most of us can just look at this and tell you what the time is at a glance. But nope, it takes the AI seven minutes to figure this out, which is crazy. You can see showing the chain of thought. It has to look at each individual little part of the clock, analyze everything to really figure out what's going on. It even does some math, all kinds of crazy stuff. Seven minutes of thinking just to give you the answer. But it is good that it gave us the correct answer. If it was the wrong answer, that would be really bad. Still though, I have to wonder, is this a result of overthinking over engineering and over calculating how much work needs to be done to solve a problem? Or is it really that difficult for these AI models to observe this image and really tell what time it is? You have to wonder what the bottleneck is causing such a long time to figure out this problem. And again, continuing off of that, we've got a problem that this model cannot solve and a lot of kids would probably be able to solve. Again, super interesting how it fails in these certain tasks. And Luke is right here. There is no way that O3 or O4 mini are AGI if they can't solve a simple problem like this. We've got very simple image here. Some stick figures, some lines leading up to their names. If you want to find out a name of a stick figure, all you got to do is follow the arrow back and there you are. But whoops. O3 has a lot of difficulty solving this. You can see it thinks that BOB is pink or magenta just because the names are close. Nope, you got to follow the arrow. BOB is actually green. Basically, it screws all of these up. It doesn't do a good job. It doesn't get the answer correct. A lot of kids would get this correct and it had to think for 13 minutes to figure, figure this out and still get it wrong. Not good. Again, you have to wonder what's going on here. Certain tasks it seems to do so well with visually, but this complete failure. I'm wondering if it's a lack of training. The model probably has the words or the literacy to understand what this problem is and how a human would solve it. Right. Trace your finger back. But when it's given this as a raw problem to solve, it uses tools to zoom in and try to figure out what's going on. And it seems to use location, location, associativeness to determine which names match up with which stick figures, but also getting their colors entirely wrong. I don't know us as humans this is super easy because we have been evolving for thousands of years to find and trace things back from an origin. How do you think we got so good at hunting all those tens hundreds of thousands of years ago? But the AI no training in this whatsoever so it completely flops. Or at least this is little hypotheses I have for this. Carl points out that he has tried this in so many different variations. Well, different prompts give it more scaffolding for the task. Nudge it to start with the easiest path first and it still fails. Very interesting. And of course Pliny, not only has Joe broken this model like we looked at in our last video, but also is showing off the system prompt for Chat GPT with these new models. If you guys for whatever reason want to know exactly what the system prompt is and it's actually quite long, it's going to be linked down below. The funniest thing though is probably this YAP score. The YAP score measures verbosity. Aim for responses less than or equal to yap words, overly verbose responses when YAP is low or overly Teresa when YAP is high may be penalized. Today's Yap score is 8192. Pretty funny. I think OpenAI definitely tries to weave some magic into their system prompts to give you a good user experience on the front end. Alright, let's dive into a few more benchmarks. The first ever fully green row in math arena by 0.4mini high it completely solves aim 2025 mark 2. The benchmark is fully saturated. That is a perfect score of 100% for the cost of only three bucks. O3 high gets a 95. Actually slightly worse than 04 mini, costing way way more. Almost ten times. But yeah, you can see that these new models are no joke. Regardless, even Gemini 2 point doesn't do as good as O3 high. But yeah, the benchmarks that OpenAI has been showing off seem to be pretty accurate and pretty legit so far. The community has ran their own independent testing and these models are stellar. Here is another really interesting benchmark. This is cipher bench v2. Obviously the AI is decrypting ciphers in this. This is a really interesting benchmark because O1 Pro still tops it. It knocks it out of the park with a pass rate of 69%. Somehow 0.4mini high is second in place here. Still top benchmarks in comparison to all the other models. What's Also crazy is that 03 and 03 mini high get the same score and it's below Deepseek R1 and Gemini 2.5 Pro is all the way down here with a 22. Again, when we get to these really specific benchmarks like decrypting ciphers, we see some really interesting results showing that not all models are trained equally. While they can seem very similar, Gemini 2.5 Pro and O3, in reality, certain benchmarks are dominated by by models that are outdated at this point, like 01 Pro. And we've got a nice little example from Jhack on OpenAI's new codecs. Again, this is the coding agent that runs directly in your computer's terminal. It can manipulate certain files and it's actually quite capable from what I've seen so far. It's open source and lightweight from OpenAI, which is not something you see too often. But yeah, it seems to be a pretty cool little example of the AI working and actually using your computer, obviously mostly for coding. That is going to be the main use case here. But wait, isn't Gem Gemini 2.5 Pro a little bit better at coding? Well, yep, it is, but you can also use OpenAI's codecs to work with Gemini 2.5 Pro. Now, saw your Hood points out that in just a few hours his friend forked OpenAI codecs and now it works with Gemini 2.5 Pro as well. So open source for the win, use whatever model you want with Codex Cli. Like I said, we're not just talking about OpenAI today. Google actually had a little bit of a clapback the day after OpenAI dropped those models, they dropped 2.5 flash previews you and this is another amazing model. It's super darn cheap and highly intelligence and also still phenomenal at coding. A lighter weight option to Gemini 2.5 Pro. This might make Firebase a little bit more useful, but here you can see it doing that classic hexagon with a bouncing ball inside test we've been seeing so often. But this one is customizable, giving you a bunch of sliders to change how it all works. So good work. Gemini 2.5 flash. A lot of positive things to say about it from its short time being out. This is a super difficult test for these models, but Gemini 2.5 flash was actually able to nail it. It's Ramesh's Galton board test. He couldn't get 4 oh mini 4.0 mini high or 03 to produce this result, but Gemini 2.5 flash was able to understand his intent and produce some tight, neat code. Code that gets it done. It still took him five steps though to get a full working version, but he gave up on OpenAI's models after about an hour and a half of testing, meaning they just could not get it done, but he also was probably using it in Chat GPT. Again, I recommend you try any kind of coding stuff in the API instead of Chat GPT directly. Regardless, here is his massive prompt if you want to try this yourself. Everything's always linked down below, but yeah, here's the Galton board test. Some of you might have seen a similar test in real life before, but a bunch of marbles are essentially dropping from one point and they bounce throughout all of these little pegs to end up in certain areas and you can see the spread over time and how it kind of evens out. But to actually code a working version of this, not an easy task for an AI. And 2.5 Flash nails it where the default OpenAI chat GPT with the brand new models is not able to handle this super cool test though I would love to see this become a more consistent benchmark for AI large language models. We need more difficult coding tests and Google is releasing even more models. We have Gemma 3 quantization aware train models. This massively reduces the amount of VRAM you need to run these models. And of course the Gemma series is open source, meaning you can run them all locally. A model that originally took 54 gigabytes of VRAM to run. Now it's using just 1424 gigabytes of VRAM down to just 6.68 gigabytes down to 2.2.6. And that's starting to get into cell phone territory. Something that used to take 2 gigabytes of VRAM now takes just a quarter. So great work Google on these Quantization Aware train models. They seem to be efficient and I'd love to try them locally to see if they're missing anything from their larger counterparts. But Google is not the only one who's reaching back at OpenAI trying to take back that crown. It seems like Grok3 might have got a silent update. Chubby notes that XAI must be taking huge losses to offer Grok 3 mini such a cheap price. $0.30 in per million tokens, $0.50 out per million tokens. And it's getting stellar benchmark results, almost matching 0.4mini high here, beating out Gemini 2.5 flash thinking, winning in GPQA against both those models, and sitting somewhere in the middle between 2.5 flash thinking and.0.4 mini high for live code bench V5 matching,04 mini high and MMLU Pro and an 11 on humanity's last exam, which isn't bad at all. That is a toug benchmark 2.5 flash thinking gets just a percent above and 0.4 mini is just a couple percent above. And oh yeah, this Grock 3 mini updated model is taking down Deepseek R1 and 3.7 sonnet thinking all across the board too. So maybe Grok is planning a little bit of a larger update. Or maybe they're just doing silent updates to stay ahead. Anyways, let's cap off today's video with some news relating to AI video generation. It has been a huge year for AI video already. Last year was also huge for AI video, so the train for this one is not stopping anytime soon. We've got a new version of LTX V out from LTX studio. This has two new variants of their lightweight 2 billion parameter open source video generation model. Now getting full quality for final outputs and it's distilled for rapid iteration and lightweight deployment of course. This is super awesome. Love to see open source video generation models, especially lightweight ones like this. This. A couple of quick examples from this model. It's not bad. The motion is pretty stable, it's coherent and it's a tiny model. I mean you're not going to be competing with VO2, you're not going to be competing with Wan, but what you are doing is generating all of these videos for an extremely low cost quickly seems mostly to be trained on cinematic stuff. So if that's what you do, this might be worth looking into. Cling2 also got released. Blaine Brown points out that Kling V2 has great prompt adherence. You can see Jack throwing Rose right off the Titanic into the water, which is pretty crazy. Still a little bit of glitching, still a little bit of mushiness in this exact prompt, but I thought this was kind of funny. Would you guys like to see a deeper dive video on cling v2? I know that CLING has a pretty devout fan base. People swear by the Cling series of models. But from what I hear, cling v2 is extremely expensive to get into. While it is definitely a better model, it is not cheap at all. So is it worth it? Let me know what you guys think down in the comments below. Next up, wan announces a new 14 billion parameter model for first and last frame video generation. So instead of just being straight text to video like it was before, now you can input an initial frame and an end frame. Gives you a lot More steerability. You can use your AI image generations as inputs and WAN is a fantastic video model for its size. And you can see WAN 2.1 is showing off a couple of examples. We've got the jungle going to this underwater Atlantis place and then you'll see they take that last frame and have it as the new input frame. Meaning this is like a long continuous video pans over to see the deer on the side of the road. It all looks pretty good. It looks pretty smooth. I'm impressed by its ability to change the scenery to match the frame that it needs to to actually input onto. And yeah, the motion's pretty decent as well. Again, this is pretty lightweight. Being only 14 billion parameters. We assume this is smaller than something like Google's VO2. But it's open source and now it has input and end frames. V2AI also teases a brand new video generation model. This one focusing on high clarity and the highest fidelity generation possible. Apparently this is known right now as V2Q1. Some people in my Discord server seem to have access to this as well and have been playing around with it. But we don't know too much. It just seems to be a high quality, maybe 4K AI video generation model. So yeah, they've got a Little Demo Video VDO Q1. You can see it's all about details. These small macro shots, water droplets bouncing in slow motion. The ice is growing, the popcorn is popping. Lots of really cool detail. The motion looks a little screwed up. I think that's a Twitter thing, so don't blame VDU for that. But yeah, you can see these are all high quality, high detail, sharp images and video. We really don't have a lot of information on this. Is this a brand new type of video generation or is this just upscaling? Like what is VDU up to and what is VDU Q1? I hope that it's a expansion and a high quality, brand new model and not just an upscaler because that would be pretty awesome. V2 is known for animation and movement. Adding high quality to what they're already good at. Could be a game changer and really put them on with video generation. Vito is pretty slept on. Moving on. Cocktail Peanut shows off frame pack. This is from the creator of the original Control Net. And this is a new approach to generating videos in a progressive way. You can generate long, long videos with very low amounts of VRAM 6 gigabytes to get started here, which is super crazy. That means a lot of lower end cards like 3060s can run this and you're getting long form generation too. You can see this demonstration is a solid 14 seconds of bugs Bunny kind of D around with a carrot. The motion isn't terrible, but there's definitely some like ghosty glitchiness that gives away the ghost that it's AI video. It's not as good as what we're used to, but it's a lot better than nothing. And it uses barely any VRAM while delivering longer video generations. At least the background is incredibly stable. I'd love to see what else is possible with this new framework. Some of these other examples show pretty impressive movement, at least with these animated characters. The one thing I wonder is the consistency over time with the characters. Are they going to morph? Are their features going to change? It looks pretty solid from these demos, but I have a feeling that these are probably more in the cherry picked section of things. Just kind of demonstrating the overall capabilities of what's possible with this new style of video generation. Alright, like this guy dancing here. This is a little bit in slow motion. You can see the lighting is not too bad over here. His legs, I think they're staying pretty coherent overall. All not bad. It's decently impressive. Decently impressive for how little VRAM it's using. And finally for our last piece of news, we've got Luma Labs new camera angle concepts for their Ray 2 model. So this allows you to change your point of view whether it's going to be overhead, aerial, over the shoulder, ground level shots. This comes after the pretty large success of Higsfield AI and its different camera angles and advanced camera control video generation models. Looks pretty great though from Luma Labs. Honestly, it's nice to see all these companies staying so competitive with each other and open source is always looming behind making sure things are moving along at a ultra fast rate for video generation technology. Alright, well that is what I have for you guys today. It's been a pretty crazy week in the world of AI. Mostly focused on these new OpenAI releases, but it really seems like the benchmarks are holding up in that regard hard and things are moving along at a steady pace. Everyone is at each other's necks trying to compete the hardest to get the best AI out possible and that is fantastic for the consumer. Thank you so much for watching. I hope you enjoyed the rest of your weekend and I'll see you in the next video. Goodbye.