EP 318: GPT-4o Mini: What you need to know and what no one’s talking about

This is the Everyday AI show, the Everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life. Chat GPT maker OpenAI just released a new model and no, it's not GPT5, it's actually called GPT4. Oh mini. But don't think small because I think this is actually going to make a huge difference. Not just on the surface because it looks like just a light version of OpenAI's latest model, but it's more than that. There's a lot more beneath the surface from a competitive standpoint. And I think this is going to greatly impact how we work, not just in a mini way. So we're going to be going over that, doing some live testing and a lot more here on Everyday AI. What's going on y'? All, My name is Jordan Wilson and Everyday AI, it's for you. It's a daily live stream, podcast and free daily newsletter helping us all learn and leverage generative AI. And this is whether you know it or not, you are going to be leveraging GPT4O mini a lot in the very near future, even if you don't know it. All right, so if you haven't already, this is one of those you got to make sure you go to your everydayai.com Sign up for the free daily newsletter. Check out your show notes if you're listening on the podcast. We always keep all that information in there to quickly get the newsletter that goes along with today's episode as well as if you want to reach out, connect with me on LinkedIn, send an email, whatever you want to do for our live stream audience. Thank you for tuning in. This is technically kind of pre recorded just by a couple hours. I'm going to be on an airplane at our normal time, so hope you can bear with me but I'll be in the comments after. Tell me what you think of this new model and for that reason make sure if you're looking for your daily deuce, daily dose of AI news, make sure to check out the newsletter also before we get started. Yeah, one quick fun announcement. Tuesday this coming up Tuesday, couple days, mark your cat calendars. That's all I'm gonna say. A thousand dollar live challenge. If you can get every single answer right in real time, there's going to be a $1,000 prize. Whether that's one person, if three people split it, they split that thousand dollar prize. If you can get every single question right on chat, GPT and other large Language models. Yeah, yeah, we're gonna do it. We're launching a little campaign called Thanks a million. So this is just a little fun event to. To celebrate that launch. So make sure Tuesday, July 23rd. If you're normally a podcast listener, Tuesday, July 23rd, 7:30am Central Standard Time. We'll have the link in the show notes as well. Enough with that. Let's get into the new model. So OpenAI just hours ago released their newest model called GPT4O mini. So this is technically a quote unquote mini or lightweight version of their most powerful. And I would say, well, not me. All tests, all benchmarks say the world's most powerful large language model right now is GPT4O. So this is the light version, right? The, the featherweight, right, the, the cheaper and more flexible and faster version of the world's most powerful model. So there are so many business use cases and so many ways that we're going to be using this in the near future. All right, so we're going to be going over all of that, but also another shout out to the newsletter. This didn't come as a surprise to me or if you listen to this podcast and read our newsletter every day. I literally told you Monday about this model GPT, GPT Mini gonna be coming soon. So make sure you read the newsletter a lot of times, y'. All. I'm a former journalist. I spend so much time researching each and every episode, even just this episode right now. Spent about two and a half, three hours reading everything, testing the model out to bring you the latest, the greatest and the truth, the realest thing in artificial intelligence. I'd say, you know, the show's unedited, unscripted. All right, so let's talk about first big picture. Let's zoom out and talk about what this means. Well, there's a lot of business use case which we're going to get into, but I even want to talk about in the competitive landscape, right, because things have changed a lot over the last couple of months, I would say notably Anthropic has been making some power moves. So Anthropic just released their Claude 3.5 sonnet. Okay, so Claude has three flavors. The smallest and fastest and cheapest haiku, the middle version Sonnet, and the most powerful version, Opus. Well, that was until they released 3.5 middle model sonnet. So they don't have 3.5 for the small one and they don't have 3.5 for the big one. And we gave you a lot of in depth insights in Our newsletter on why. That's why you got to read it. All right, so anyways, also we saw Google just also recently come out with their 1.5 flash. So again a small version. So. And we've been talking about this on the show as well. I've been saying for a long time the future of large language models is smaller models. And it seems like everyone else has been playing there, right? Anthropic has been playing there, right? With Haiku specifically. That's their small fast model. They just announced last week that you are also now able to fine tune that model on aws Amazon Web Services, right? So a lot of flexibility being able to fine tune a model like that. We're going to go into more what that means, don't worry. But also the same thing with Google 1.5 Flash just announced I believe early, I think it was late late June, early July. So very recently in the last couple of weeks, everyone else has been making this shift toward small models. OpenAI hasn't until today. So people are only talking about the mini part, right? They're only talking about, oh, this is just a smaller version. This is a crappier version of GPT4O which a lot of people, I don't understand it, right? If you're a model geek like me, you might appreciate this. Everyone else you might not. But everyone's making These complaints, oh, GPT4O is worse than the previous version, GPT4 Turbo. No, it's not. People. I think people have this unrealistic expectation, right? Like when you get something set up for like GPT4 Turbo and then you try to use it exactly as is in GPT4. Oh, and it might not work as well. Well, for a reason, right? You have to re engineer whatever you built, right? You can't argue. I don't understand people that argue with math and science and benchmarks. GPT4O is the most capable model by far. It's not even close. Look at the benchmarks, right? All right, so anyways, big competition here and there's a lot of kind of upward mobility for these other companies, right? Especially Anthropic when they are going to release a 3.5haiku and a 3.5opus. I mean it's going to go wild. All right, so I'm going to try to keep my dorkiness at bay here, but this is why this is ultimately important. All right, so I have a screenshot for our podcast audience. So shout out Tom Keldenick on, on the Twitter machine here. He kind of broke all this down for us, uh, to look at the pricing because this is ultimately what it comes down to. This isn't just a small version, a faster version of GPT4O. It is a cost efficient and very capable version. So up until recently it was Claude Haiku that had the kind of most affordable in terms of power. All right, so there's, there's a, a quadrant that we will show. I, I, I should have thrown it up here for the live stream. I can probably find it, you know, if I can multitask. Okay. But you know, up until recently I would say it was Haiku and probably Gemini Flash that had the right kind of balance of cost and output quality. Right. So when you were looking at a small model that's ultimately all these businesses, thousands, tens of thousands of businesses are using OpenAI's should have started by explaining this, right? So many of the, of, of the systems, of the software, of the tools that you use on a daily basis, you might not even know it because it's happening behind the scenes. The future of just about everything is generative. AI is large language models. And so many of these companies that you use softwares that, that, that you subscribe to things that, social media, everything, right, they're running behind the scenes on large language models. And what that means in most cases is they're using an API from a company like Anthropic Claude, like Google Gemini, like OpenAI. So that's why this cost and output balance is so important because like it impacts our daily lives and we don't even understand it. So before today, you know, if we look at input in outputs, okay, so I'm not going to go into tokenization, but think of tokens kind of like words. They're parts of words, parts of phrases. Right? We've gone through, I've had a whole episode on tokenization, so I'm not going to take up your time. All right, so this is the price per million tokens. All right, so $0.25 input for Claude Haiku and a $25 output comparatively. Now we have GPT4O mini, $0.15 on the input versus Claude Haikus, $0.25 cheaper. All right, marginally. And then we have the output $0.60 for a million tokens versus a $25 haiku. And for comparison, GPT4O, the big boy model. Right, let's do that. $5 per million input. So again, $5 versus $0.15 and then on the output 15 for GPT4O versus $0.60 for GPT4O mini. All right, so now you can probably hopefully understand why that balance is so important. Right. The balance of power and affordability. And, and you know, it's. We're going to show you some, some examples here of why I think this is striking the right balance. Right. All right, so let's keep this thing going and let's go over some of the basics here. All right, so some of the basics of the model, it's more cost effective. We're going to talk about benchmarks, but it got a very impressive 82% MMLU score. So again, this, this classification of models, right. We're not comparing it to Opus, we're not comparing it to, you know, the most powerful models out there. These are the small, large models, right. I don't know if these are technically considered small language models. I think for that we be looking at, you know, Google Gemini Nano and some of the Mistral models. Right. So these are technically think of them as small, large models. I know that's a little confusing. We also don't know how many parameters it is. However, it scored an 82% on the MMLU score. So we're going to talk about a little bit more on those benchmarks here. A little bit. But that is an extremely impressive score for a small, a smaller model also on some instances on the new leaderboards. Right. So we talk about the Chatbot arena leaderboard all the time. It is already out benchmarking GPT4 Turbo, the previous version before GPT4O. That is extremely impressive, especially when you look at the cost differentiation. That's what I'm saying. It's the cost. It's the cost. It's the cost. And what that means. Well, you know, the output too, right? You have to have that cost. Output, sweet spot. But it is the cost in the output that is going to really change things. All right, a couple other things we talked about $0.15 per million input, $0.60 per million output, and it is more than 60% cheaper than GPT 3.5 Turbo. Also, this is replacing 3.5 Turbo. All right, that's also important to keep in mind because so many applications, developers, companies started to build a backbone for their future business off of GPT 35 Turbo. Right? So this is another important thing to keep in mind here. All right, so now I'm going to go ahead. I kind of talked about this sweet spot, right? So I'm going to go ahead now for our live stream audience. I'm going to go ahead and share my screen. I'll try for our podcast audience to describe this. The best I can okay, so this is essentially, you know, you have your X axis, your Y axis, a little quadrant, right? So price, the more expensive is on the right and then quality is down. So you don't want to be writing down. That means you're expensive and not good quality. So right now you kind of have GPT 35 Turbo there. You have Command R some other models. Okay. But you want to be in the upper right left hand quadrants. Okay. That means you have the highest quality at the cheapest price. So is GPT4O mini the absolute cheapest when it comes to, you know, price per million input and output? No. Mistral and Llama are a little bit cheaper, but the quality of those is much, much lower. I mean, an MMLU score, I don't think people understand. An MMLU of 82 is wild for this price. Wild. Right. And I'm getting excited about it because I read these like scientific papers like on the weekend. I'm a dork and I've been reading them for years. Right? So even getting something in the 80s, like a year or two ago was, was mind blowing, but expensive. The fact that it is this cheap, if you are a business leader, if you are looking to figure out what is your future in a generative AI world, this changes it. This changes it. It's not even close. All right, so that's kind of an overview of the model. And again, we talk about this significantly decreasing costs. All right, so this is a little, little graph here tweeted out by the open AI developer account. And it just shows a downward step in the price over time. Right? The fact that In March of 2023, so about what, 15 months ago it was $2 for a million tokens, now it's 24 cents. That's a blended. That's a blend right there. Just, just FYI, that's a blend of input output going from $2 per million tokens to $0.24. Right? It's a 90. Reduction in the quality is astronomically better. Things are getting cheaper and they are getting better, but faster than anyone ever could have predicted. Right? The very famous interview with Bill Gates and Sam Altman on Bill Gates podc, that was one of the biggest takeaways was they were both flabbergasted by how fast the development has come and how, how much cheaper essentially all these models are because it is cheaper to train them as well. Right, that's, that's another piece of the puzzle, right? When we talk about, you know, chip makers, GPU makers like Nvidia, they're Becoming more and more efficient and in turn more and more affordable, which drives everyone's costs down. All right, let's keep this thing going. Let's talk about some capabilities and features of the new GPT4O model. All right, so it does support text and vision in the API. It's also important to note this is accessible on the front end of Chat GPT. I'm going to show you all of that here in a minute. So this is accessible via the front end of Chat GPT. If you log into your Chat GPT account, you can use it. There's. I don't think there's a lot of use cases if you're using it on the front end. There's a couple that we'll talk about, but for the most part, if you're just logging into Chat GPT and using this as a front end user, you're probably not going to be using this new model a lot. However, if you are a developer, if you're working on the back end, GPT4O mini is your new best friend. It is amazing. It is going to change the way that you can do business, right? Because essentially, and this is just an explanation, if you are a beginner here, like myself, I'm kind of a beginner, but I'm kind of not. But you know, for, for the everyday person, like, why is Jordan talking so much about the cost and the training and all of this? Well, again, like I said, this is going to change how all, almost all companies online, you know, everyone's AI enabled, right? Go on. Anything, everything on the tv, everything's AI, AI, AI. Guess what? They're all going to be paying significantly less now, which ups the competition in the marketplace. But also in theory, for consumers, we can start demanding lower prices or there will be competitors that come up that can provide better quality at lower prices. Right? If the quote unquote power or if, if the cost of electricity goes down by 90% and you're in the business of using electricity, guess what, that changes your business. All right, so it does support text and vision in the API and in the chat interface. So it features support for text, images and coming soon, video audio inputs and outputs. More on that in a bit context window in the API, not in the chat interface is 128,000 tokens. All right? It supports up to 16k output tokens per request and same knowledge base, same knowledge cutoff is October 2023, as well as an improved tokenizer for non English text. In other words, it converts the inputs for non English Better to tokens. That's the thing a lot of people don't understand when we talk about tokens. Go listen to that episode. But essentially, ChatGPT doesn't know words. Both when you give it words and it gives it back. It converts everything to tokens to actually understand what you're talking about. Right. So the tokenization in other languages is very important. All right, last but not least, here in our bullet points, benchmarks and performance. So let's just go straight into the chart for this one. Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you, you can partner with us too. Just like some of the biggest companies in the world do. Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen. If you're on the podcast, the benchmarks are silly. They're, they're, they're silly like never, even if you follow large language models. Did someone a year ago, two years ago could look at this chart and think that this would ever be possible at this price? Okay. GPT4O mini is outperforming all other small, quote, unquote small models except for the Math Vista. All right, so essentially we have what, eight different. Yeah, we have eight different benchmarks. So from the MMLU to the drop to the MGSM Math Human eval. Mmmu. Right. So all these different benchmarks that all researchers put models through, you get a score that tells you, is a model good or is it not? Is it a bunch of marketing or does it actually have the chops? This is the thing I don't understand, y'. All. When people are like, oh, GPT4O is, is so, so much worse compared to GPT4 Turbo. No, it's not. You just have to re look at what you're doing. Right? Large language models are generative. That's the thing people don't understand. Just because you had something working in a previous model doesn't mean it's going to work in the next model. Get to, get to know it. You're. You might have to re engineer some things. Are some things worse? Absolutely. Across the board. No. You can't argue with more than a million votes in the chatbot arena leaderboard. All right, so some of these worth noting, some of these metrics. I just want to point out a couple. So mmlu, that is the massive multi task language understanding test. So that is I would say still the gold standard. I think we're probably going to be moving to the MMMU as the gold standard because that's multimodal. But the think of this as text based. Right? MMLU is essentially the ACTS for large language models. It is the gold standard. What is your score? Right? And it measures the model's knowledge across 57 different subjects, including STEM, humanities, social sciences, everything. Right. Y', all. This blew everything out of the water. It blew out Gemini Flash, Claude haikyuu, Claude Haiku GPT35 Turbo. It is not close, right? You might look at it on this chart and be like, oh, it's only a three point difference behind Gemini Flash and about a what? A seven point difference behind Claude Haiku. That in mmlu, that is a lifetime away. When we look at the biggest models, you know, they're usually within like a half point, half point, maybe even a point one one I believe opus. You know, Claude, three opus when it first came out was like 0.1 higher than GPT four turbo. Right. Usually it's like a half point to be three to seven points ahead of your nearest small model competitors means that this is freakishly good. I cannot under understate that how much that little seemingly small margin means in the mmlu. Same thing in math purpose blows everyone out of the water. We're talking like 30, almost 30 points ahead of the others. Wow. So math purpose is essentially complex math topics such as calculus, linear algebra, higher level mathematics. So it is freaking really good at math. All right. And that's important for data analysis. Again, we have to think not just what you are going in to do inside of chat gbt. That's not what this is about. Where this is going to pay dividends and change the way we all work is through the API is through those hundreds or hundreds of thousands, right. Of different companies, startups, products that you use, you know, you're probably playing with a bunch of AI tools all the time. I have dozens of AI tools that I use on a week to week basis. Mostly all of them use GPT. They use the, the GPT4 from open AI. Right. So this is going to change and improve the quality of all these other tools that you use, especially if you're someone in marketing, advertising, communications, or if you're just someone who's constantly playing with AI tools in large language models. This changes everything. All right, all right, now we're gonna have some fun, y'. All. Foreign, let's go ahead and do a couple of things live. All right, so a while back, let's see if we can get a good, good view here for our live audience. And hey, podcast crew, I'm going to do my best to describe what we have going on here. Pretty, pretty simple stuff. All right, so about, about a month or so ago, when Claude 3 Opus or Sorry, Claude 35 Sonnet came out, we did some comparisons. We just did some, some live prompts and we compared the output. Some of them were deterministic. Right? There's a right answer, there is a wrong answer. Some of them required us to make a call. All right, so now what I'm doing, I am in the open AI playground. Okay, so, so what that means is kind of the back end. If you build something, you know, all developers who are using the API, this is where they go. But I'll tell you this, you don't have to be a super technical person. You can get a free account, you can go in there and just play in the playground. It's a great way to learn. If you want to become better at prompt engineering, if you want to become better at understanding models, do this inside of the playground. Anyways, so I'm comparing two different models side by side at the same time. So we're going to be looking at, we're going to be asking some questions that have a simple yes and no answer. Is it right or is it wrong? And then we're probably going to be doing some that require a little bit of. Let's use our brains. Okay, so again, I did this previously between Claude35OPUS and GPT4O. So now we're going GPT4O on the left, GPT4O mini on the right, doing the same prompts. We're going to be looking at speed, we're going to be looking at quality, and we're going to be looking at, in some cases, a little bit of reasoning and logic. All right, so the first one again, if you've tuned into the show, we've done these a couple of times. We're not changing the system prompts. We're leaving everything as is. So out of the box. Here we go. The first one. I just woke up today with six apples and three bananas. Yesterday, I ate an apple and. Or, sorry. Yesterday I ate a banana and two apples. This morning, I will eat one apple and no bananas. However, I don't really like apples, and one banana may turn brown tomorrow. Assuming nothing else changes, how many apples and bananas will I have tonight? All right, we're going to hit run, and we're going to see who finishes first. And do they both get it right? Wow. Okay, so GPT4 oh, mini was faster there. Pretty impressive. All right. And let's see if they got it, who got it right and who got it wrong. Interestingly enough, GPT4O got it right. O. Mini got it wrong. There we go. We already have one use case. This is one. I've seen similar ones like this. I kind of made it up, you know, or modified something similar that I had already seen. So it looks like. Let's see where GPT4 oh, Mini kind of got tripped up. So it got tripped up on something that I put in there to trip models up. I test models all the time. Yeah, I told you guys, I'm a geek. All right, so I put something in there about eating. Eating the fruit yesterday. But guess what? I started it off by saying I just woke up today with six apples and three bananas. So whatever I talk about yesterday is irrelevant. So GPT4 oh, Mini did not understand that GPT4O got it right. The correct answer is five apples and three bananas. And GPT4O. Mini said three apples and two bananas. So he got tripped up by yesterday. Hey, if I give you today, yesterday doesn't matter. All right, let's do another one. Kind of. Kind of a tricky one. We're going to. I think for the most part, I'm always going to refresh. It shouldn't. It should. Well, actually, I'm not. It's going to take too long because we're not going to be worried about the context here. All right, the next one here, I'm saying if it takes three hours to dry 10T shirts in the sun, how long will it take to dry 30T shirts in the sun? All right, so let's go ahead. Click, run. This should be a quick one. All right, so this one GPT4O was faster. Oh, mini was slower. So the opposite last time, oh, mini was faster. All right, the correct answer should be three hours. So let's see. GPT4O. Got it right. It said it will also take three hours. Let's look at four. Oh, mini. Let's see. And it says it will still take three hours. Okay, so our score, our score so far for those keeping, keeping track. So we're going to do fast and we're going to do right and wrong. All right, so so far we have. It's. It's tied in fast, but in the end, fast doesn't matter if you're wrong. Right? So one, we have many, and one, we have O. And then. Right, so right now we have. We're gonna do this. We're gonna do right. Sorry, y'. All. I want, I want to make sure for our podcast audience we can, we can recap this at the end. All right, so so far we have. We have O. Right, Sorry, I'm making a little chart here. Oh, right, Got two and then we have many. Right, so far, just one. All right, next prompt here. This is pretty. Well, this one is. It takes a little logic here. All right, so here we go. And what I do love is at the bottom, you can see the latency or how long it took, and also the tokenization. All right, so the next prompt here, a box is locked with a three digit numerical code. All we know is that all digits are different. The sum of all digits is 9, and the digit in the middle is the highest. What is the code? So I've even gotten tripped up on this and models have gotten tripped up. But before I hit enter, there's actually multiple answers. One time I got confused and a model said this is the only correct answer. And I'm like, huh, yeah, that's right. And no, there's actually multiple answers. So let's go ahead and run it and see how it goes. All right, so both are going pretty fast. Still computing. All right, so they finished just about at the same time. It looks like mini was just a little bit faster. But let's see who got it right. So I'm looking here at GPT4O. It's walking me through things. It says the digits are different. The sum of the digits is 9. The middle digit, it is the highest. So it's doing a little bit of a formula. And let's see, it's. So it's going through different cases. Case one, case two. So it says, I don't even know if either of these finished generating. They may have stopped. It looks like they both Stopped halfway and didn't complete. So let's. I'm going to try this one more time and see if they finish this time. Okay. So it doesn't look like GPT. I'm double checking here. It doesn't look like GPT4O finished this, which generally when I do this, it finishes. I don't know if it's because I'm in the compare mode and doing head to head. It really shouldn't make a difference. So it's going through. Yeah. I'd have to read this. I'd have to read this. I'd have to take a little time. There's a lot of math on the screen. It didn't give me a clear answer. I'm going to try the prompt and I'm going to say at the end, I'm going to say, please respond only with the correct answer or answers. I'm going to say, you do not have to show your work. Sorry, I'm getting confused. Too much math on the screen. It's late. All right, here we go. That worked a little better. So neither of them got it right. We'll just say both. Okay, so neither. Neither got it right. So GPT4O said the code is 273. Middle digit is the highest. However, those do not add up to 9 when combined. And GPT4O mini said the code is 1, 4 7. Also do not add up. So they both got that one wrong. All right, we're gonna do just one more test. Maybe. Maybe we'll do two more. So I'm going to do one. I'm going to say, please let me get. Let me get the. This other prompt that I had ready to go here. So this one is about kind of starting a company. All right, so let's go ahead. And here it is. I did this before. This one's a little longer. So I'm going to click run while they go and I'll look at the latency because this one's a little longer. Okay, so what I'm saying here is create a new company and brand for a future smart home device. This will solve a problem that does not currently exist. To start, come up with the company's name and its first flagship product. Give the product a new branding campaign, go to market strategy, tagline and rationale for why it'll work. Respond in a succinct way, keeping responses to short bullet points, but with ultra specific facts. All right, so from a speed standpoint, it looks like GPT4O was slightly faster, but not by a Lot. All right. And there is no right or wrong here. This is subjective so I'm not going to read the whole thing. So let's just see the takeaway here with what GPT4O came with. It came up with Verity Solutions. The flagship product was nomisy. Not sure what that is. It's eliminating the complexity and stress associated with managing and memorizing the various names and access code for different smart devices in a home. All right, okay, I see that. Let's see. The branding campaign is smooth. Management starts here. I got some social media campaigns. Simple. Go to market strategy in good. Multiple phases. Good. The tagline is no mezzy. Simple. Okay. Not the best, but not, not terrible. Right again. How, how I wanted the output. Right. I said keep it succinct, give it to me bullet points. So I think this is where in theory GPT4O may have shined over GPT4O mini. Anyways, let's look at GPT4O mini, what it came up with. So it is Home Vigil and the flagship product is Lifeguard. So again I'm asking it to solve a problem that does currently not exist in a future smart home device. All right, so GPT4O mini came up with Home Vigil, company name. The flagship product is Lifeguard. It is a smart home device equipped with AI driven sensors to detect subtle changes in a household's atmosphere such as smells, noises, movements etc to predict potential safety hazards. All right, that's pretty good. All right. The branding campaign is Safety in the Silence. Oh, okay. I like that from Lifeguard. Safety in the Silence. Same thing. Go. Go to market strategy. It even gave us a target audience which I don't think that GPT4O did. Gave us different channels. All right, I like that. Gave us even a launch event. Whereas for the Most part on GPT4O it just went through phases. It was awareness, engagement, expansion, where on the other hand, GPT4OH Mini really took it a little more granular. It gave us a target audience, different channels, you know, different strategies for those different channels. A launch event. So pretty good. And the. Oh, I just realized y' all why the. Why we didn't get the. The full from the code one. It's because I think I have a. A cut off here. That's. That's probably, that's probably what it was. That's probably what it was. All right, so anyways, I think for g. For this test, I would say GPT4.0 mini won this one. All right, so I'm, I'm Actually, I'm cranking back up the maximum tokens. I feel I didn't give a fair shake. I. I kept wondering, I'm like, why is this cutting off when we are asking about the locks? So we're going to do that one one more time. But however, I'm giving that one. If I'm being honest, I'm giving that one to Mini. I didn't expect Mini to, To win in something that, you know, I'm not saying this involved logic and reasoning, but it was much better. Right. GPT4O gave me something kind of complicated again, large language models are generative. I'm. I'm pretty confident if I did that a hundred times, GPT4O would win out against many. But that just shows many is capable. If you don't think it is, that just shows to me. That shows me it's capable. All right, I'm gonna run this, this code one one more time. That's why it was cutting off. I forgot I. I have that setting on there. You can restrict how long the output is so you don't accidentally spend way too much money. Right, because you're paying for the tokens, but they're pretty cheap. All right, so, wow. Okay. A lot here from GPT4. Oh, Mini. And we still just have more math code problems. So here we go from. Okay, so we got a. We got multiple correct answers from gpt4o when we reran this. So essentially it needed more memory. That's all it is. It couldn't do it with the limited tokens. It actually wasn't done. I cut it off early. So GPT4O got multiple correct answers. 153. That's correct. Middle digits highest. Adds up to 9. 163 is not correct. That adds up to. That adds up to 10. So it did get one correct variation. 153. All right, so let's go ahead and look at GPT4O mini. So it says one valid code is 243. That's correct. So that adds up to nine. So it didn't give me any wrong answers. GPT4O. Mini. GPT4O gave me one wrong. There's actually a lot more correct answers. And GPT4O one time gave me a list of like, I don't know, like 15 correct answers. So in this case, actually, GPT4O mini got it a little more right because it didn't give me a wrong answer. I'll give that one like a half. It didn't give me any wrong answers. It only gave me one correct answer, whereas GPT4O gave me a correct answer and an incorrect answer. So, all right, let's do one more. And this is our last one. This isn't supposed to just be a live head to head comparison, but I wanted everyone to be able to get at least a good understanding of this. So for this one, here's what we're going to do. We're going to test the vision capabilities. So I just dropped a photo. I took this on the road in Chicago, all right. And I'm putting in a prompt. So this one, I am pretty curious which one's going to finish first. So I'm saying please identify where this picture is located, what direction the photo is facing, and every other detail that you can make out. All right, so I'm going to go ahead for our live stream audience. I'm going to throw this up on the screen big so you can look at it. I got, I'm telling you, I got the perfect photo to use for this because it shows like, I don't know, 20 cars, but you can't even see any license plates. So it's not like I'm even putting anyone's, you know, personal data out there. I was kind of happy about that one. However, the correct answer is we're on 9094 heading southeast. And you can tell that by. So this requires actually a lot of knowledge, which is why I use this. Because the way that you can tell this, aside from the order of the buildings. Right, that's one way. Because you have the Chicago skyline and depending on which way you're looking at it, right. If you're looking at it like north, straight down, the buildings, the skyline are going to be in a different order. If you're coming up from the south, they're going to be in a different order. Right? So that's one way. But also you can see two different exit signs. So as an example, we have 46A California, 47A Fullerton. So in theory it should know, right? It's, it's heading in this direction. The, the, the numbers are going up. Right. So it has plenty of clues. All right, so let's jump in, let's run this live and see how each model does. Let's go. All right, so put the prompt in GPT4O much faster. Mini took a while to get started. Okay, interesting there. So MINI took longer to get started, but was actually done much faster. GPT4O got started way quicker, but took a little bit longer. But ultimately, what I care about is who got it correct. All right, so, interesting. So the GPT4. Actually, let's keep going left. Left to right here. Great job. GPT4O. This is. This is very impressive. All right, so it says the photograph has taken in Chicago. Illinois landmarks in the background. Willis Tower, John Hancock Center. Cool. It says direction. The photo is facing southeast towards downtown Chicago. Correct. Specific highway details. It's breaking them all down. Right. 47A for Fullerton, 47B for California yield sign. It's talking about traffic. It's even telling me it's probably during rush hour. It wasn't. It was on a Sunday. I think there was a Cubs game. Anyone else a Cubs fan? Are we Sox fans? Who's your favorite baseball team? Let me know. Infrastructure. This is great. It's giving me a huge break. It's even going into vegetation. Y', all the Vision models are nutty, like a squirrel on keto. All right, Weather and time. My gosh, this is good. It's giving me guesses. It says general vibe. It says typical urban highway. This is good. So it answered everything correctly. Not only did it answer the simple question about where I was and what direction the photo was taking, but I also asked it give me every other detail that you can make out. So, interestingly enough, GPT4O. Mini's vision doesn't give me a lot of. It doesn't give me a lot of the same. It essentially says I can't identify specific locations or details about the image you provided. It does say there's traffic, there's cars, there's tall buildings. You know, it. It shows that there's Fullerton Avenue and California Avenue. There's photos in the sky. So that's interesting. Right? So actually, what I'm gonna do. So hands down, I'm giving. I'm giving. Oh, the win over on this. All right. Many did not win. GPT4O won this. So I'm gonna say. I'm gonna add. Add some more things. I'm gonna put. Please take your time. Go step by step and understand the context clues in order to provide the answers. All right, so I'm wondering with a little more, quote, unquote, prompt engineering, Right? Yeah. With a little more prompt engineering, I'm wondering if GPT4OH Mini can do this, because I was assuming it could have. Again, this is available in the API, so if you're thinking, oh, should I be tapping into the API? I mean, it didn't get anything wrong, but it also didn't get it right. Or am I just really underestimating or not giving enough props to the level of detail that GPT4OH gave. All right, so I'm doing it one more time. I'm just doing a rerun. Just. I'm curious if a little bit of extra prompting will help. GPT4O mini. So, same thing. GPT4O got started right away. Looks like we're getting stalled out. Looks like we're getting stalled out here on GPT4O mini. Yeah, server errors. I'm. I'm not surprised, y'. All. This just came out. People are building on it. They're. They're breaking it. I'm actually surprised how fast it's been, if I'm being honest. GPT4O mini. Considering probably all dorks in the world are trying to break this thing right now, including myself. We're going to try it one more time, see if. See if we're going to get another error message. But, hey, I'm curious. Podcast audience, drop me a line. Are you going to be using this? Are you going to be moving everything over to GPT4O mini? What tests are you running? All right, so here we go. Same thing. GPT4O did a great job. Let's see if GPT4O mini did a little bit better. So it did. With a little bit more prompting and a little bit more help on my end. It did say Chicago. It noticed the Willis Tower. Although. What's the Willis Tower, y'? All? That's Sears. Don't mess. Now it's. Now it's giving me more information. Traffic situation, road signs, direction. So let's see if it got the direction. The direction the photo is facing appears to be toward the city skyline, suggesting the photographer is moving toward the downtown Chicago. Okay. Correct. It didn't get it. So I'm going to try one more time. Again, we're giving the win on this one to GPT4O. I'm going to say, given all of those context clues, what direction is the photographer facing? All right, GPT4 already got this. Right. I'm just wondering if we can squeeze the right answer out of GPT4. Oh, mini. Gosh. I mean, GPT4. Oh, super impressive. All right, let's see. So it said, given the context clues, particularly the presence of Chicago con line in the background, the photographer is facing east toward downtown. Yeah, it's east. It's southeast, but got it right. So with better. That's y'. All. That also goes to show, number one, understanding generative AI. It's a roll of the dice. You're Gonna get something different if you go in there with a weak prompt. I went in there with a weak prompt. The, the point of this wasn't prompt engineering, but when I followed it up with a little bit better prompt and I followed it up with a secondary, you know, when I prompted iteratively or just followed it up with hey, now let's try it again. In this way, it eventually got it right. All right, so that wraps up our live look. Now I'm going to quickly go over what no one is talking about. I know this is a longer one, but I cannot underestimate the importance. So a couple things. If you are on a free account, it's actually hard to access this right? On the front end of Chat GPT, they actually changed it. So previously you had a traditional model switcher. Now all you have is, and this is brand new as of today, I believe now you just have Chat GPT, which is if you're on the free plan, that's what you're on. And then it says Chat GPT plus. So, oh, I forgot my other, my other slide here. But essentially you can still access. Let's let me go ahead and share, share my screen here and show you. So you can still access the updated version here. So what you have to do, first you have to prompt, right? And then after you prompt, then you get the new model selector. So if you actually want to try out GPT4 mini and you are on a free plan, you can't select it on the normal drop down. You have to first do a prompt and then there is a model switcher right underneath chat GPT's response. And then you can do chat GPT mini or sorry, GPT4 oh, mini. So now I'm on mini and it's obviously asking me is this better, the same or worse? All right, I'm not here to train you right now, model. I'm in the middle of a podcast. All right, so number one thing you need to know, it's a little hard to access for free users on the front end of Chat gbt, which I think a lot of people are. So that's why I wanted to throw this out there. This is everyday AI for everyday people, right? Number two, something people aren't talking about. What does this mean for the future of Apple's plans? Right? Because number one, people aren't talking about this fact either. Apple is actually. They built their own large language model. They didn't even mention it by name. Man, I feel bad for all those software developers at Apple during Apple's WWDC in June. They didn't even mention their own large language model by name. They didn't even really explicitly in the keynote, at least I'm talking about in the keynote, they didn't even explicitly decipher that. Yes, when we're doing all of this Apple Intelligence quote unquote, that there's actually two different models. We have our own Apple on device edge AI model running essentially all the queries that just have to do with the information on your device and for most everything else, you're going to be prompted to use GPT4. I, I believe GPT4. Right. But what does this mean now? A much smaller model? Again, we don't know how much smaller it is. I'm sure we'll find this out soon. I don't think that this model will be small enough to fit on a future, on a future device, right. At least a phone. I think right now, you know, the Google smartphones, the S24s that actually have a large language model living locally on the phone. That's why it's important, right. There's more privacy, it's faster, it's more energy efficient. Right. So I think the future is working edge AI, right. On device smaller models. But I still think that this new model from OpenAI GPT4O mini is probably too small to run locally. But I would think the next version of it, probably, whether that's in six months or a year, is probably going to be able to fit. Which is wild to think about, right? Which number one, the impact on the environment. Right. The more edge AI that you can run, the better the environment is. Right. One of the biggest problems right now is this uses a lot of power. A single prompt to chat GPT takes up 10 times more power than a Google search. Right. And as we're using large language models more and more, you have to think of the environmental factor. So moving to edge is better for everyone. Moving to on device is better for everyone. It's faster, it's more secure, it's better for the environment. But right now it's not really possible for a lot of use cases, right. To get the power that you want, you know, squeezed on a, on a, on a little device, not possible. But I think probably in the future it's going to be maybe the next version of GPT4OH mini might be able to run locally on a smartphone, which would be amazing. So you have to look at the future of this Apple and open AI partnership. Here's another. Aside from cost, there's actually A reason you might want to use GPT 4 oh mini I talked about, hey, if you're on chat GPT, you probably aren't going to want to use it because there's real no advantage. You're not paying for it. Anything differently in Chat GBT, right there. If you're on the Chat GPT plus plan, you're paying 20amonth and so you don't have to worry about cost. The main advantage here is cost in the API. If you're building a third party application, piece of software, whatever, you know, building fine tuning your own model for a company with, with retrieval, augmented generation, right? But here's the other key benefit that no one's really talking about. It is the first model to apply OpenAI's brand new, which we just talked about on the show this week, instruction hierarchy method, which helps to improve the model's ability to resist jailbreaks, prompt injections and system prompt extractions. So I mean that, right? There is a great reason in why I think so many companies are going to be flocking to GPT4O mini, right? You want your business to be safe, right? The last thing you want is prompt injections for someone to jailbreak, right? A lot of companies don't take proper safety precautions, right? And part of it is because it's so easy. It's so easy to essentially create a wrapper, right? Put in a little bit of your company's data, do a little fine tuning. You know, it's not super expensive like it used to be. It's almost so easy that too many companies are doing it and they don't take proper safety measures. So this new version of GPT4O mini in their API will be a godsend for those companies that are maybe facing some of those issues. All right, Another advantage or sorry, another thing no one's talking about. And why is no one talking about this? I should have started this show out with this. Ready? Do people not like reading? I still like reading. This was toward the bottom of OpenAI's blog post announcement. So they said today, GPT4OH Mini supports text and vision in the API with support for text, image, video and audio inputs and outputs coming in the future. Oh, look at that. I believe this is the first time in writing that we've seen open AI said in the future there will be audio and video input output. Sora, right? Yeah. Sora accessible via an API. Sora accessible via chat GPT, presumably. That's huge. And also video input. Did you guys know that you can actually input a Video into chat GPT right now go try it. See, tell me, tell me what happens. So this is huge though. Again, this is all reasonable. Assumptions pointed to this, right? Yeah, you'll be able to. It's. It's multimodal, right? The future is multimodal. That's the whole point of the GPT4. Oh, which is Omni. Instead of using three, technically three separate models on the hood, it's using one. So this was always the assumption, but I believe this is the first time in writing that OpenAI said, hey, input output, you're going to have, you're going to have video input and output. So pretty big, right? Right now that's actually one area Gemini's been crushing it in with their super long context window. Being able to upload videos and it can scarily know everything. Yeah, talk about wild use cases of generative AI. All right, number five, more and better AI is coming everywhere. So like I said, you probably don't even know this, right? As an example, maybe the wealth management company you use has a little AI advisor that you talk to, or maybe a software that you use for work has a customer support bot powered by GPT. Right? You probably don't realize it. Apps on your smartphone, so many things use GPT behind the scenes, all right? And so many of them are about to get so much better, right? Which is related to number six, the prices are going to be. The prices are right now, but they're going to be getting so affordable companies can't afford not to build on generative AI. Let me say that again. Prices for building on top of a large language model are getting so affordable companies can't afford not to build on it, right? If you would have asked me a year and a half ago, hey, should all companies be building on top of large language models? I said it very early on in the podcast, probably like 13 months ago. I said no, companies shouldn't. Because look at the prices back then and the quality wasn't good. The quality wasn't good and it was too expensive. Now, my gosh, quality is great and it is cheap. And think now, right? If your company is, is on the fence today, well, first of all, reach out to us, right? We offer consulting services, we have other partners we work with. We can walk you through that if you're not sure, right? I'm lucky enough I get to talk to super smart people all the time. I just had a conversation today with, by the time you guys hear this yesterday with, you know, a senior director at Microsoft AI, I'm Lucky enough to get to talk to literally the smartest people in the world, building things. So we can help walk you through this. So reach out to us. You know, I have my, my email, my LinkedIn in the podcast and you know, you guys know how to get me here on the live stream. But companies need to be building. If you haven't made that decision now look at this visual on the screen. This step down in price, right? And then think of the increase in quality. You need to be thinking now, okay? You have to be thinking. So it's, it's getting it right now where you need. And this, this is going to sound bad, but you need to start comparing the costs to train, deploy and upkeep a model versus the cost of humans. Because it's going to be lopsided soon. It's going to be lopsided, right? That's why I think everyone is going to be working. Not everyone, but so many people are going to be quote unquote working in AI, right? Taking, you know, this first, I call it first party company data and turning that into knowledge for your model. All right, let's see if we can beat the hour mark here. Number seven, Energy demands are gonna be bonkers. That's my last thing you need to know. I kind of already alluded to that. But all of these things adding up, these models getting faster, cheaper, better, easier to work with. Demand is going to be. I, and I'm not just saying OpenAI GPT 4 oh mini. Because guess what comes next? Anthropic is going to strike back. Google's going to strike back, which is going to make open AI strike back. Energy demands are going to be bonkers. Until we can get all of these models or many more of these models running locally, energy demands are going to be off the, out of the roof. And, and so, and at the same time, watch Nvidia stock, watch AMD stock, watch Qualcomm stock. All the companies that are making chips. Obviously Nvidia has an unfair head start. Their stocks are just going to continue to rise in the long run because we all need the gpus. Right? So that's it in, in depth. Look, I hope, I hope if you stuck with me to the end, let me know. I, I always, I'm always curious, did you make it to the end? I don't know. Send me an email or DM on LinkedIn with the word pancakes. I'm just curious, did you make it to the end? Let me know. And I'm hungry for some pancakes. So I hope this delivered. And I hope now you better understand GPT4O mini, because like what I talked about in the beginning, it's actually not mini. It's actually, I think a pretty big deal. This is a pretty big deal. I think this is going to greatly change the way that we all work and really the future of generative AI. All right, thanks for tuning in, y'. All. If this was helpful, please consider reposting this Leave us a rating. If you're listening on Spotify or Apple, appreciate the support. And don't forget Tuesday, July 23rd. Hopefully you're listening, you're watching this. Before that time, mark your freaking calendars, y'. All. 7:30am Central Standard Time, July 23rd. The link for the LinkedIn live stream. Hopefully LinkedIn doesn't go down. If so, you can join on YouTube. You got to get your question, you got to get your answers in live. All right, A thousand dollars to anyone who gets a perfect score. If no one gets a perfect score, no one gets the money. There's going to be some easy questions, there's some going to be some trick questions. But if you are an avid listener of the Everyday AI show, you're probably have a decent chance that you're going to get most of these right. You got to be quick, but you got to be there. So please join us and make sure to go to your Everyday AI dot com. Sign up for the free daily newsletter. Thanks for tuning in. Can't wait to see you next week and every day for more Everyday AI. Thanks y'. All. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.