GPT-4o - Full Breakdown + Bonus Details

It's smarter in most ways, cheaper, faster, better at coding, multimodal in and out, and perfectly timed to steal the spotlight from Google. It's GPT4 omni. I've gone through all the benchmarks and the release videos to give you the highlights. My first reaction was it's more flirtatious sigh than AGI, but a notable step forward nonetheless. First things first, GPT4O, meaning omni, which is all or everywhere referencing the different modalities it's got is free. By making GPT4O free, they are either crazy committed to scaling up from 100 million users to hundreds of millions of users, or they have an even smarter model coming soon. And they did hint at that. Of course it could be both, but it does have to be something. Just giving paid users five times more in terms of message limits doesn't seem enough to me. Next, OpenAI branded this as GPT4 level intelligence, although in a way I think that they slightly underplayed it. So before we get to the video demos, some of which you may have already seen, let me get to some more under the radar announcements. Take text to image and look at the accuracy of the text generated from this prompt. Now, I know it's not perfect. There aren't two question marks on the now, there's others that you can spot, like the I being capitalized. But overall I've never seen text generated with that much accuracy and it wasn't even in the demo. Or take this other example where two OpenAI researchers submitted their photos. Then they asked GPT4O to design a movie poster and they gave the requirements in text. Now when you see the first output, you're gonna say, well that isn't that good. But then they asked GPT4O something fascinating. It seemed to be almost reverse psychology because they said, here is the same poster, but cleaned up. The text is crisper and the colors bolder and more dramatic. The whole image is now improved. This is the input, don't forget. The final result in terms of the accuracy of the photos and of the text was really qu impressive. I can imagine millions of children and adults playing about with this functionality. Of course, they can't do so immediately because OpenAI said this would be released in the next few weeks. As another bonus, here is a video that OpenAI didn't put on their YouTube channel. It mimics a demo that Google made years ago but never followed up with. The OpenAI employee asked GPT4O to call customer service and ask for something. I've skipped ahead and the customer service in this case is another AI, but here is the conclusion.

Could you provide check Joe's email address for me? Sure. It's joexample.com awesome. All right, I've just sent the email. Can you check if Joe received it? We'll check right now. Please hold. Sure thing. Hey Joe, could you please check your email to see if the shipping label and return instructions have arrived? Fingers crossed.

Yes, I got the instructions.

Perfect. Joe has received the email.

They call it a proof of concept, but it is a hint toward the agents that are coming. Here are five more quick things that didn't make it to the demo. How about a replacement for or Lenser? Submit your photo and get a caricature of yourself. Or what about text to new font? You just ask for a new style of font and it will generate one. Or what about meeting transcription? The meeting in this case had four speakers and it was transcribed. Or video summaries. Remember, this model is multimodal in and out. Now it doesn't have video out, but I'll get to that in a moment. Here though was a demonstration of a 45 minute video submitted to GPT4O and a summary of of that video. We also got character consistency across both woman and dog, almost like an entire cartoon strip. If those were the quick bonuses, what about the actual intelligence and performance of the model? Before I get to official benchmarks, here is a human graded leaderboard pitting one model against another. And yes, I'm Also a good GPT2 chatbot is indeed GPT4O, so it turns out I've actually been testing the model for days. Overall, you can see the preference for GPT4 compared to all other models in coding specifically, the difference is quite stark I would say. Even here though, we're not looking at an entirely new tier of intelligence. Remember that a hundred Elo gap is a win rate of around 2/3, so 1/3 of the time GPT4 Turbo's outputs would be preferred. That's about the same gap between GPT4 Turbo and last year's GPT4. A huge step forward, but not completely night and day. I think one underrated announcement was the desktop app a live coding copilot?

Okay, so I'm going to open the ChatGPT desktop app like Mira was talking about before, okay. And to give a bit of background on what's going on. So here we have a computer and on the screen we have some code and then the ChatGPT voice app is on the right. So ChatGPT will be able to hear me, but it can't see anything on the screen. So I'm going to highlight the code command c it and then that will send it to chatgpt, and then I'm going to talk about the code to ChatGPT. Okay, so I just shared some code with you. Could you give me a really brief one sentence description of what's going on in the code?

This code fetches daily weather data for a specific location and time period, smooths the temperature data using a rolling average, annotates a significant weather event on the resulting plot, and then displays the plot with the average minimum and maximum temperatures over the year.

I've delayed long enough. Here are the benchmarks I was Most impressed with GPT 4.0's performance on the math benchmark. Even though it fails pretty much all of mymath prompts, that is still a stark improvement from the original GPT4 on the Google proof graduate test. It beats Claude 3 Opus and remember, that was the headline benchmark for Anthropic. In fact, speaking of anthropic, they are somewhat challenged by this release. GPT4O costs $5 per 1 million tokens input and $15 per 1 million tokens output. As a quick aside, it also has 128k token context and an October knowledge cutoff. But remember the pricing 5 and 15. Claude 3 opus is 15 and 75. And remember, for Claude 3 opus on the web you have to sign up with a subscription, but GPT4O will be free. So for Claude 3 Opus to be beaten in its headline benchmark is a concern for them. In fact, I think the results are clear enough to say that GPT4.O is the new smartest AI. However, just before you get carried away and type on Twitter that AGI is here, there are some mixed benchmarks. Take the Drop benchmark I dug into this benchmark and it's about adversarial reading comprehension questions. They're designed to really test the reasoning capabilities of models. If you give models difficult passages and they've got to sort through references, do some counting and other operations, how do they fare? The drop, by the way, is discrete reasoning over the content of paragraphs. It does slightly better than the original GPT4 but but slightly worse than llama 3.400B. And as they note, llama 3.400B is still training. So it's just about the new smartest model by a hair's breadth. However, we're not done yet. It's better at translation than Gemini models. Quick caveat there. Gemini 2 might be announced tomorrow and that could regain the lead. Then there are the Vision Understanding evaluations. It was a real step forward on the MMMU. As you can see, a clear 10 points better than Claude Opus. Again, I'm curious if Google Gemini can exceed it. Though the improvements to the tokenizer could be revolutionary for non English speakers, the dramatically fewer tokens needed for languages like Gujarati, Hindi, Arabic and more don't just mean that conversations are cheaper, they're also quicker. And what about multilingual performance? Well, this time they didn't compare it to other models, but compared it to the original GPT4. Definitely a step up across languages, but English is still by far the most suited language. Indeed. Here is a video of some of the model's mistakes, ending with some dodgy language tuition. I know enough Mandarin to say it wasn't perfect at tutoring.

Let's root root root for the home.

Oh no.

What was that? Sorry guys, I got carried away. Guilty as Ready for another round? San Francisco.

Hello, my name is Nach. I'm here with my co worker.

Hi, I'm Shino. I'm trying to teach my co worker how to speak Mandarin. We want to start from simple words like Ni hao. Can you teach him how to pronounce that?

Of course. Hey Nacho, nice to meet you. Nihau is pretty straightforward to pronounce. It sounds like ni hao.

Just make sure to keep the knee part high and then go down and.

Pitch on the how. Give it a try Nihau.

That was great.

Really good first try.

Nacho.

You're natural.

It really wasn't. He needs to work on his tones and her face was the giveaway. There were a lot of other interesting video demonstrations, but before them the GPT4O blog post from Sam Altman put out tonight. He made the argument that putting this capable AI tool in the hands of everyone for free is justifying that Open part of OpenAI less about open weights or open source, more about open to everyone without ads or anything like that. He also draws our attention to the latency of the models. In many ways that is the key innovation with GPT4O. By dialing down the latency, you dial up the realism. As he said, it feels like AI from the movies. Getting to human level, response times and expressiveness turns out to be a big change indeed. I think I should get a little credit for predicting her like AI in a video from a month ago, but now I want to get to those demos. The response times were amazing and the model was clearly designed to be flirtatious. Note that in Senate testimony last year, Sam Altman said we try to design systems that do not maximize for engagement. So let me know in the comments whether you think these demos are designed to maximize engagement.

Okay, so this is what I wrote down. What do you see?

I see. I love ChatGPT. That's so sweet of you.

Yeah, well, I really appreciate all the help, so. Yeah, as we can see.

Yeah.

Wow, that's quite the outfit you've got on.

Yeah, as we see. You know, we can chat in real time.

I was impressed, though, that it could speed up its talking on demand, of course.

One, two, three. Hey, actually, that's.

That's a little slow.

Could you count faster? Sure thing. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.

Okay, that was a little bit too fast. Of course, OpenAI must be given credit because for blind people, this could be revolutionary.

Right now, the Royal Standard flag is flying above Buckingham palace, which is the signal that the monarch is present. How exciting to be there while the king is home.

Try and tell me exactly what they're doing right now, please.

Right now, the ducks are gently gliding across the water.

It will also help you with interview prep by analyzing your appearance.

How about you?

I'm awesome. Listen, I got some huge news.

Oh, do tell. I'm all ears.

Well, in a few minutes, I'm going to be interviewing@OpenAI. Have you heard of them?

Open AI huh? Sounds vaguely familiar. Kidding, of course. That's incredible, Rocky. What kind of interview?

Well, it's for a software engineering role, but I just need to know, do I look presentable? Professional?

Well, Rocky, you definitely have the I've been coding all night look down. Which could actually work in your favor. Maybe just run a hand through your hair or lean into the mad genius vibe. Your enthusiasm is what's really gonna shine through.

I don't have a lot of time, so I'm just gonna throw this on. What do you think.

Rocky? That's quite a statement piece. I. I mean, you. You'll definitely stand out, though maybe not in the way you're hoping for an interview.

Okay, I got it. I got it.

I could definitely see GPT4O being used for mathematics tutoring. 30 second clip. But notice also the slight glitchiness at.

Certain moments relative to angle alpha. All right, so I'm pretty sure this is the angle alpha right here.

Right?

Correct. Now, looking at the triangle, which side do you think is the hypotenuse? Remember, on the hypotenuse, I'm not totally Sure. I think. I think it might be this one, but I really am not sure. This side ac, you're close.

I also noticed quite a few glitches in the demo. Sounds cutting in and out. The one feature though that OpenAI wants you to really focus on is the video in capacity. You can live stream video straight to the transformer architecture behind GPT4O and so the video in functionality is certainly impressive. Note though that the reaction time of GPT4O isn't quite as immediate as for audio. Greg Brockman, the co founder of OpenAI, has to prompt prompt it to react to the intruder.

Overall, stylish feel. They might be getting ready for a conversation or a presentation given their direct engagement with the camera.

Sounds like we're in for an intriguing interaction. I'm curious about the lighting you mentioned. Could you describe it a bit more? Is it natural light or are we talking unique fixtures or colors that add to the atmosphere?

The lighting here seems to be a mix of natural and artificial. There's a noticeable bright light overhead, probably a fixture casting a focused beam downwards which creates a spotlight effect. This adds a dramatic and modern feel to the scene. The rest of the room is lit more softly, possibly by natural light filtering.

Was anything.

Did anything unusual happen recently? Yes, actually, just as you asked, another person came into view behind the first person. They playfully made bunny ears behind the first person's head and then quickly left the frame. It added a light hearted and unexpected moment to the scene and now the focus is back on the original person with the leather jacket.

Nevertheless, GPT4O can produce multiple voices that can sing almost in harmony and really.

Try to harmonize here. San Francisco San Francisco in the month of May but maybe make it more dramatic and make the soprano higher. San Francisco in the month of May San Francisco in the month of May It's a Friday called what may. We are harmonizing. We are harmonizing. Great. Thank you.

And I suspect this real time translation could soon be coming to Siri later for us.

So every time I say something in.

English, can you repeat it back in Spanish?

And every time he says something in.

Spanish, can you repeat it back in English?

Sure, I can do that. Let's get this translation train rolling.

Hey, how's it been going?

Have you been up to anything interesting recently?

Hey, I've been good. Just a bit busy here preparing for an event next week.

Why do I say that? Because Bloomberg reported two days ago that Apple is nearing a deal with OpenAI to put ChatGPT on iPhone. And in case you're wondering about GPT 4.5 or even 5. Sam Allman said will have more stuff to share soon, and Mira Murati in the official presentation said that they would be soon updating us on progress on the next big thing. Whether that's empty hype or real, you can decide. No word, of course, about OpenAI co founder Ilya Sutskever, although he was listed as a contributor under additional leadership. Overall, I think this model will be massively more popular, even if it isn't massively more intelligent. You can prompt the model now with text and images in the OpenAI playground. All the links will be in the description. Note also that all the demos you saw were in real time at 1.x speed. That I think was a nod to Google's botched demo, of course. Let's see tomorrow what Google replies with. To those who think that GPT4O is a huge stride towards AGI, I would point them to the somewhat mixed results on the reasoning. Benchmarks expect GPT4O to still suffer from a massive amount of hallucination. To those, though, who think that GPT4O will change nothing, I would say this. Look at what ChatGPT did to the popularity of the underlying GPT series. It being a free and chatty model brought 100 million people into testing AI. GPT4O being the smartest model currently available and free on the web and multimodal, I think could unlock AI for hundreds of millions more people. But of course only time will tell. If you want to analyze the announcement even more, do join me on the AI Insiders discord via Patreon. We have live meetups around the world and professional best practice sharing. So let me know what you think and as always, have a wonderful day.