OpenAI's new GPT-4o Mini costs $0.50 per million input tokens and outperforms Gemini Flash and Claude Haiku on reasoning and math benchmarks. The video walks through setting it up in Google Colab with sample queries, showing how cheap it is to run per call.
Chapters
All right, so today is July 18th, 2024, and OpenAI has just released their most cost-efficient small model called GPT-4o Mini. We have GPT-4 already, but this is the mini version of GPT-4o. And so I'm just going to go through this document real quick, highlighting the most important things, and then you can read it for yourself. So I'm going to gloss over it real fast. The main things we really need to take away from this is some of the scores that it got from the key benchmarks that it was benchmarked against. So we have reasoning, which is for textual intelligence and reasoning benchmarks. It scores an 82% on MMLU benchmark. And so of course, take this with a grain of salt. Some of this MMLU benchmark, a lot of people have questions about it or trust issues with it as well. So that's one other time. But you can see compared to Gemini Flash, you had 77.9 for Gemini Flash, and then for Cloud Haiku, 73.8. So it 82%, it's pretty good. And then we have math and coding proficiency, which also here it scored 87%, which is really impressive if you ask me. And then 75.7% for Gemini Flash, 71.7% for Cloud Haiku. And so the other part here as well is multimodal reasoning, and it shows strong performance on this benchmark MMU, and it scored a 59.4 compared to 56 0.9 for Gemini Flash and 50 for Haiku. So not too much of a big difference there, but still an improvement for the size that it is. And so this is kind of like the overall benchmark of all the benchmarks that it's been put there. So you can see, like, for example, math, it does very well compared to the other models. And of course, the purple one is GPT-4o, just the GPT-4o that you've kind of gotten used to. And then the orange one is the GPT-4o Mini. So you can dig into that and, you know, and read more about it. But Some of the highlights here as well, there's built-in safety mechanisms, of course, that's also to be talked about. And then we have availability and pricing, which is probably what you're interested in right now. So it is available now as a text and visual model in the Assistants API, Chat API completions batch, yada, yada, yada. And so all that. And so one of the key things here as a developer, you only pay $0.50 per 1 million input tokens. When you type in your prompt or you want to feed in your prompt, you'll be charged $0.50 for 1 million tokens that you input, and then $0.60 per 1 million output of token that you get back from the API. And so they say that roughly equivalent to 2,500 pages in a standard book. So that's pretty big. And fine-tuning will be rolled out in the coming days if you're interested in fine-tuning it. And they did post here It's kind of a comparison. I think the chart makes a little more better kind of clear visualization. So GPT-4, GPT-3.5 Turbo, I started here, it was $2 per 1 million tokens. And then 3.5 Turbo 0.613 was $1.60 per 1 million tokens. So you can see there was a drop there. And then there was another also drop here from 0.163 to 1.106. And that came to $1.20 per 1 million tokens. And then it dropped significantly here as well with the next Turbo release to $0.70 per 1 million tokens. And then GPT-4 Mini to $0.24 per 1 million tokens. So this is really interesting because it seems we're just running all the way down to zero at some point, right? Because we've dropped quite a lot here. So the next one should be like literally a couple of cents, which is good for us as developers because we get to build more with less money spent. OpenAI Playground, and the best way I found it, you can come here, you can pick your model, you could pick GPT-4 Mini, you can insert your prompt here, and then you can chat with it there. So that's cool and all, but of course you want to go ahead and use the API outside of this playground. So one easy way you could do it, come here to the code. Once you've picked the model, you can see example boilerplate, copy that, and then you can go to something like Google Colab, which I have here as an example to fire up for us to test it. So the first thing you want to do is install OpenAI. So you run that. That should be good. I think we already ran that. Then the next step here is really importing that OpenAI as a package. And then you'll need your API key, which you would've gotten from your OpenAI account. And then from there, this is where I would save it. So you come to secrets on this tab on the corner, the notebook, write in the name, kind of this, think of it as a variable. So OpenAI API key, and then you put in the API key right there. And then you have to turn it on so that the notebooks can access it. And then That's really it. And then after that, if you want to use that API key within your Colab, you copy this boilerplate code here, which is really importing user data and then being able to export import that from the secrets there. And so that's what we did here. So what I did, I instantiated the client OpenAI API key, passed in that user data and then API key, which is the variable there, and then that loaded in the API key. And then I created a function here with Chat completion model, GPT-4o mini, and then system prompt here. I added it as you are helpful assistant respond in respond in markdown format. You can write whatever you want there, and then the prompt I passed in as a prompt because we'll take the prompt here in our function, and temperature one balance. It could be the max is two, and then one is kind of the middle point. This kind of like just making it more creative there, and so max tokens I put it at a thousand top one of course, and then after that. We return the response. And so here's an example. So let's run this piece, and then we'll run this piece as well. And so we can just ask it a few questions. How many Rs are in the word strawberry? I can't type strawberry to save my life. So strawberry. So the answer should be three, of course, right? But let's see what it does. Oh. Look at that. It contained 2 Rs. Yeah. No, you failed. It's 3. All right, let's ask it maybe another question. Write a snake game, Python snake game. And we are not gonna run the code cuz we are in Jupyter Notebook. We could do that for another time. This is just supposed to be an overview video of what, how you can load it and use it in your API and just kind of a quick test, early quick test. And so the goal for this was for, for you to get more information about this real fast. So here is the code that it generated and I can't say it works cuz I haven't tested it. But as you can see, you know, suddenly, you know, actually installed in Python and then it tells you what to do there. And then, you know, initializing imports. So this is pretty good. I think, I think it looks good. We haven't tested it, but I'll give it a test as well. And we'll see how that works. And then, you know, cut off here because the token limit that I set probably ended here, 1,000 tokens. So it didn't get to the point where it gave us all the full instructions. This is today's. And for GPT-4 Mini, I've ran a couple of runs already and it's less than 0.1 cents. Let's see. Let's refresh it and see if it if it goes up a little bit more. But unless it doesn't update in real time, so yeah, it's still less than a cent. Just wanted to give you a quick rundown on that and how that works, and catch you in the next video.
