GPT-4o Mini and the Rise of Smaller, Low Cost AI Models

Today on the AI Daily Brief, we're talking about OpenAI's new GPT4O mini model and why it says a lot about where AI development and competition is happening. Before that in the Headlines Google is bringing AI to the Olympics the AI Daily Brief is a daily podcast and video about the most important news and stories in AI. To join the conversation, follow the Discord link in our show. Notes welcome back to the AI Daily Brief Headlines edition. All the AI Daily news you need in around five minutes. We have the Olympics coming up pretty soon. They are kicking off on July 26 in Paris and Google will apparently be the official AI sponsor for Team usa. So what does that actually mean? Well, it means a few things. First of all, the broadcast will pull from the immersive views that were added to Google Maps over the last few years, adding additional information and giving 3D views for the venues where the competition is happening. Announcers and commentators will use Google Search AI overviews in broadcast segments, for example, answering trivia questions about the Olympics. In one part of it, comedian Leslie Jones will ask Gemini to help her learn a new sport, and so on and so forth. This is a straight up advertising partnership. This is a chance for Google to show off what it's got, which could go either very well for the industry or quite poorly. For now, we will just have to wait and see. Over in Chipland, the information is reporting that OpenAI is holding talks with Broadcom about developing new chips, Summarizes Yahoo Finance OpenAI is exploring the idea of making AI chips on its own to overcome the shortage of expensive GPUs that it relies on to develop models. The report says that OpenAI is hiring former Google employees who worked on Google's Tensor Processing unit, and that they've decided to develop an AI server chip. An OpenAI spokesperson was circumspect, saying Open OpenAI is having ongoing conversations with industry and government stakeholders about increasing access to the infrastructure needed to ensure AI's benefits are widely accessible. Which basically reads to me like them saying, yeah, of course we're having conversations about building our own chips. Everyone is having conversations about building their own chips. I think for now it's not clear exactly what will come out of those conversations or where any actual partnerships would come down. Next up There has been a lot of discussion this week of Samsung's AI features on their new Galaxy Z Fold 6. The Verge, for example, writes, the Galaxy Z Fold 6 sketch to image tool is ridiculous, fun and slightly worrying. They write. Samsung would very much like us and its shareholders to know that its new phones are the AIest phones that ever AIed, and the Fold 6 that I'm testing comes with a new tool called Sketch to Image. Draw a rough sketch on a photo or an empty note page and it will use generative AI to turn that into an image. I shrugged it off as just another AI thing when Samsung announced it on stage, but it's really good. So good that it worries me a little. Using the Sketch to Image tool in a note is pretty harmless. You draw something, highlight it and choose from a variety of styles like 3D cartoon and illustration and turn your doodle into something more detailed. The author says that they worked with their two year old to draw things like goofy looking dump trucks and school buses. But then they say using Sketch to Image on a photo is where things got weird. The problem, they said, is something they called the B problem. I took a photo off a dock just south of downtown Seattle with some flowers in the foreground. Because they're close to the camera and my focus was in the distance, they're slightly blurred. I drew the world's worst sketch of a bee on one of those flowers, figuring AI would insert an in focus image of a bee, giving it away easily as a fake. Wrong the issue, the author writes. If I didn't know the bee's origin story, there's no way I'd think twice about it. If I scrolled past that on Instagram, I'd assume the photographer snapped the picture at just the right time and hung around waiting for a bee to fly into the frame. To be fair to Samsung, there is a little watermark that says AI generated content down at the bottom, but it would be very, very easy to miss. The author did note that the results aren't always that good. For example, a giant pirate ship in the harbor is probably not tricking anyone. Ultimately, for this author, there are questions but no good answers and ultimately just a recognition that the world we're moving into is very new and unpredictable in how it will all play out. Analysts also pointed out that Apple's AI strategy and Samsung's AI strategy could not be more different. Yahoo Finance writes Samsung is angling to quickly build a large user base for its generative AI services, thus incentivizing developers to build apps for its Galaxy AI plat. To do that, they're making those AI features available on phones from the last three generations. Overall, they write, Samsung says it expects to put Galaxy AI on some 200 million devices by the end of the year. Apple, meanwhile, will basically only give the very most recent generations of the iPhone access to Apple Intelligence. That's because, of course, Apple is trying to use this to boost iPhone sales. They want to use it as a way to get people to upgrade rather than holding onto their quote unquote good enough iPhones. Whether that works will likely have a big impact on how Apple stock performs at the end of this year. Lastly, today Google has announced a new AI industry forum called the Coalition for Secure AI, or COS AI. They write the new industry forum will invest in AI security and leverage Google's secure AI framework. Their first three work streams are software supply chain security for AI systems, preparing defenders for a changing cybersecurity landscape, and AI security governance. Founding members of the organization include Amazon, Anthropic, Chainguard, Cisco, Cohere, Genlab, IBM, Intel, Microsoft, Nvidia, OpenAI, PayPal, and Wiz. And all of this will be housed under Oasis open, the International Standards and Open Source Consortium. On the one hand, it feels to me like there are so many of these announcements that it's never exactly clear which ones are actually significant or not. But the fact that all of the big players are here, with the notable exception of Meta, suggests that this might be a fairly significant effort and one that we will keep an eye on. For now, though, that is going to do it for today's AI Daily Brief Headlines edition. Next up, the main episode. Today's episode is brought to you by Superintelligent, the platform for fun Fast AI learning Super has a ton of new things going on. We recently announced our partnership with Spotify, through which users of that app can now access super intelligent content directly from their mobile apps. We've also just launched the AI Learning feed. In addition to seeing the tutorials that we're dropping, there are polls, news items with related lessons, and a chance for people to show off the projects and use cases that are making AI come alive for them. We've also just kicked off the Super Summer Challenge, where each week we'll share a new challenge that you can use to discover new AI tools and use cases. Go to BeSuper AI and use code SUPERFUND for 50% off your first two months. That's BeSuper AI. Today's episode is brought to you by Venice. The leading AI companies store your entire conversation history and attach it to your identity forever. Every question you ask, every answer you receive, every image you generate, every thought you share with the machine, it's all being spied on. If you trust all the companies, hackers and NSA board members that will ever have access to your AI conversations, then rejoice for you are well served. For the rest of us, Venice is an alternative. Venice is a powerful AI app for text, image and co generation that respects you as a sovereign individual and believes privacy and free speech are not only human rights but are necessary for civilizational advancement. Private, permissionless and uncensored. You can try it for free without an Account at Venice AI welcome back to the AI Daily Brief. Today we are talking about GPT4OH Mini and it is both more powerful and cheaper than GPT 3.5 Turbo. Now, in addition to discussing GPT4OH Mini specifically, we're also going to be talking about the changing nature of competition among models, price decreases among models, and what it all means for the state of AI. But first of all, let's talk about this announcement. The way that Sam Altman, CEO of OpenAI teed it off was saying towards intelligence too cheap to meter and cost, as we will see really is the theme in their announcement post. They write OpenAI is committed to making intelligence as broadly accessible as possible. We expect GPT4OH mini will significantly expand the range of applications built with AI by making intelligence much more affordable. GPT4 mini scores 82% on MMLU and currently outperforms GPT4 on chat preferences in LIMSYS Leaderboard. It's priced at $0.15 per million input tokens and $0.60 per million output tokens, an order of magnitude more affordable than previous Frontier models and more than 60% cheaper than GPT 3.5 Turbo. OpenAI also gives a sense of what types of new use cases this low cost opens up, including applications that chain or parallelize multiple model calls, applications that involve a full code base or conversation history, and applications that interact with customers through what they call fast real time text responses like customer support chatbots. Currently the model supports text and vision, with full multimodal support for text, image, video and audio coming. It's got a 128k token context window as well as supporting 16,000 output tokens per request. And basically they trying to point out is the best performing small model. It scores higher on the MMLU than Gemini Flash or Claude Haiku. Same on the MGSM which measures math reasoning. On that it scores much higher. They also point out that it scores much higher on coding performance, which seems to be one of the use cases that they're most focused on. Before releasing this broadly, they say they partnered with companies like Ramp and Superhuman, who they say found GPT4OH mini to perform significantly better than GPT 3.5 turbo for tasks including extracting structured data from receipt files or generating high quality email responses when provided with thread history. Some of the response was lukewarm. Professor Ethan Malik writes, first impressions with GPT4O mini is that it is impressive for a small model, but no replacement for a frontier model. When given complex education prompts, it can't follow instructions as well and misses nuanced GPT 4.0 nails i4 success, writes point of view when you were waiting for GPT 5, but GPT 4.0 mini is coming and it shows a guy punching a laptop. They continue. OpenAI is on a mission to piss everyone off. Everard's Pietro Scarano writes, never seen such a lukewarm reaction to an OpenAI model release. Sean Ralston, who does API Dev support at OpenAI Pietro this release is about price and performance. GPT 3.5 was no longer competitive against Gemini, Flash, Cloud, Haiku, etc. Many business customers using LLM APIs for routine functions like company chatbots and standard work need and want low token costs. GPT 4.0 mini sets a newest standard of affordability with close to frontier model performance. Of course we all want smarter models forthcoming, but those tend to require more compute and therefore are more expensive. Pietro responds, I agree on all these fronts, just sharing the overall sentiment. From my standpoint, however, the response has been exactly focused on what it seems like the important thing was, which is cost. Ben's byte summed this up by saying OpenAI's 4.0 mini brings big brains on a budget. It's a small model that beats the early version of GPT4 and it costs 30x less than its big bro GPT 4.0. They sum up. You can now say bye to GPT 3.5 Turbo as 4o mini will take its place in ChatGPT's free tier. If you were using GPT 3.5 Turbo API in your applications, you should switch to 4point mini. That leads to two reasons to care people are getting increasingly smarter AIs for free directly via ChatGPT and app developers can build powerful AI tools without breaking the bank. And really, I do think that cost is the story here. The end of OpenAI's blog post reads the cost per token of GPT4OH Mini has dropped by 99% since text DAVINCI003, a less capable model introduced in 2022. Vittorio says 99% reduction in cost in two years is crazy and that cost reduction has big implications. Back to that blog post again they write, we envision a future where models become seamlessly integrated in every app and on every website. GPT4OH Mini is paving the way for developers to build and scale powerful AI applications more efficiently and affordably. The future of AI is becoming more accessible, reliable and embedded in our daily digital experiences. Vittorio Even jokes, in 2026, OpenAI will give you money to use its models. Professor Ethan Malik came back again and said the big story with GPT4 oh mini is its cost and how quickly the price of intelligence of a sort has dropped. Now this is the point where I want to return to that now infamous Goldman Sachs report Gen AI too much spend, too little benefit. This is from June 25th and I've already done a full takedown of this. But one of my biggest critiques is the argument from Goldman Sachs head of Global equity research Jim Covello that he thought it was unlikely that the cost of AI was going to come down. The interviewer asks, even if AI technology is expensive today, isn't it often the case that technology costs decline dramatically as the technology evolves? Part of Jim's response was to say the tech world is too complacent in its assumption that AI costs will decline substantially over time. Now, the point that he was making in this section is that he didn't think there was going to be enough competition for state of the art GPUs to actually bring those costs down. What he seems to have missed is that we're officially in a time where the AI that we have right now is performant enough that there are many applications that don't require the absolute state of the art, but instead just require near state of the art. And those costs are coming down and coming down dramatically. What's more, it's not even clear the costs of state of the art aren't coming down dramatically as well. Claude 3.5 Sonnet, which is Anthropic's most intelligent model and which many people have switched entirely, including me basically from GPT4 2, does also represent a cost reduction. Claude 3.5 Sonnet, for example, costs 1/5 of what Claude 3 Opus does. Andrej Karpathy wrote LLM Model Size Competition is intensifying backwards. My bet is we'll see models that think very well and reliably that are very, very small. Two parameters for which most people will consider GPT too smart. The reason current models are so large is because we're still being very wasteful during training. We're asking them to memorize the Internet, and remarkably, they do and can recite Sha Hashes of common numbers or recall really esoteric facts. Actually LLMs are really good at memorization, qualitatively a lot better than humans, sometimes needing just a single update to remember a lot of detail for a long time. But imagine if you were going to be tested closed book on reciting arbitrary passages of the Internet given the first few words, this is the standard pre training objective for models today. The reason doing better is hard is because demonstrations of thinking are entangled with knowledge in the training data. Therefore, the models have to get larger before they can get smaller because we need their automated help to refactor and mold the training data into ideal synthetic formats. It's a staircase of improvement of one model, helping to generate the training data for the next until we're left with a perfect training set. When you train GPT2 on it, it will be a really strong and smart model by today's standards. Just maybe the MMLU will be a bit lower because it won't remember all of its chemistry perfectly. Maybe it needs to look something up once in a while to make sure. This certainly reflects what I'm seeing, and what's more, I think it's Responding to a business demand There have been numerous posts recently about how enterprises are getting more sophisticated in their needs. With AI. They realize that yes, there are some cases that they need the absolute state of the art for, and they need it immediately when they need it and are willing to pay for the fastest best dancers. But there are tons of functions that don't need to be immediate and that don't need the absolute state of the art. The Information recently published a piece called why Smaller Could Be Better, they write. While some developers are racing towards super intelligent AI technology, others are focused on building cheaper, more practical models. Over the past six months, several big tech companies, including Google and Microsoft, have released small language models in a bid to stake their ground in a burgeoning area of AI research. These models are lightweight enough to run on phones instead of on the cloud. They generally have fewer than 3 billion parameters, a tiny fraction of the more than 1 trillion parameters believed to support OpenAI's GPT4. And this is a whole additional case that we haven't talked about, the fact that everyone is racing for on device AI. The reality here is that the lower the cost, the more different types of applications can be built with AI in it. And that really seems to have been the motivation OpenAI president and co founder Greg Brockman wrote, we built GPT 4.0 mini due to popular demand from developers. We love developers and aim to provide them the best tools to convert machine intelligence into positive applications across every domain. It's hard to overstate how big a difference it makes to reduce costs by this much. Latent Space's Swix, for example, writes, if you're interested in LLMs for summarization, my evaluation of GPT 4.0 Mini is out. TL Dr. Mini is the same or mildly worse in some cases, but because it's 3.5% the cost of 4.0, I can run 10 versions of the Mini and use Mini 4.0 to judge and still have money left over to donate. Point being that we are just at the beginning of seeing how many more applications AI will come to as costs come down, and I think that's incredibly exciting. For now, though, that will do it for today's AI Daily Brief. Thanks for listening or watching. As always, and until next time, peace.