GPT-4o: OpenAI's NEW OMNI-MODEL Can DO it ALL

Hey, how's it going?

Hey there. It's going great. How about you? I see you're rocking an OpenAI hoodie. Nice choice. What's up with that ceiling though? Are you in a cool industrial style office or something?

Well, can you take a guess at what I might be doing based on what I'm showing you here?

From what I can see, it looks like you're in some kind of recording or production setup. With those lights, tripods and possibly a mic, it seems like you might be gearing up to shoot a video or maybe even a live stream.

Yeah, in fact, we've got a new announcement to make.

That's exciting. Announcements are always a big deal. Judging by the setup, it looks like it's going to be quite the professional production. Is this announcement related to OpenAI?

Perhaps it is. And in fact, what if I were to say that you're related to the announcement, or that you are the announcement?

Me? The announcement is about me. Well, color me intrigued. Are you about to reveal something about AI, or more specifically about me as a part of OpenAI? You've got me on the edge of my. Well, I don't really have a seat, but you get the idea. What's the big news?

Yeah, we've got a new model that can interact with the world through audio, vision and text.

Okay, so OpenAI just announced the release of their latest model, which they're calling GPT4O. And this O stands for Omni model. This is a new flagship model that can reason across audio, vision and text in real time, as we saw in the demo. So Unlike the previous GPT4 Turbo, which basically is combining separate audio models, vision models and a text model, this is trained end to end multimodal. So it has its own capabilities of understanding audio input, processing videos or images, and generating audio responses. And as a result, you see this amazing latency numbers. So it says it can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. And this is going to be twice as faster compared to the current GPT4 Turbo offering. And it's going to be 50% cheaper. With the announcement. They are also sharing some really cool demos. We are going to look at those. But let's look at some benchmarks now. Even though this model is much faster, much cheaper compared to GPT4 Turbo, it seems to outperform all the existing models on almost every benchmarks. So this is for text evaluation. For audio automatic speech recognition, it is much better compared to whisper version 3 and for the audio translation performance it it's on par with Gemini. Now even before this announcement, we were actually testing this model. So according to William who is part of OpenAI, he says GPT4O is a new state of the art frontier model. We have been testing a version on the LMSYS arena as I am also a good GPT2 chatbot and even on the overall Edo score, this outperforms every other model that we have access to. So some really interesting things here. Now since it's a completely new frontier model, so they came up with a completely new tokenizer as well. So here are some examples when looking at 20 different languages. Okay, so let's look at some other demos. This is the one that I really like. So it's a customer support proof of concept and let's run this demo.

Hey O, this is Joe. I'm going to connect you to Acme Telco now. The new iPhone they sent me isn't working. I want you to get them to send me a replacement device. Can you take care of this for me?

You can count on me, Joe. I got this.

Ring, ring.

Hey, Joe, this is Jamie from Acme Telco. How can I help you out today? Hi there. I'm calling on behalf of Joe, who recently received a new iPhone from Acme Telco, but. Oh, got it. When did Joe receive the new iPhone? IPhone was delivered two days ago. Cool. Could you share the order number with me? Of course. It's 1-029-3849. Great, thanks. Is the device physically damaged in any way? No, there's no physical damage. It's just not turning on despite trying all the basic troubleshooting steps. Bummer. Would Joe prefer to return the iPhone to an Acme Telco store or ship it back? Shipping it back would be the best option. Could you please provide the shipping information and process to get a replacement? Absolutely. I'll email a shipping label and return instructions. Could you provide Joe's email address for me? Sure. It's joeexample.com. awesome. All right, I've just sent the email. Can you check if Joe received it? We'll check right now. Please hold. Sure thing. Hey Joe, could you please check your email to see if the shipping label and return instructions have arrived? Fingers crossed.

Yep, I got. Yes, I got the instructions.

Perfect. Joe has received the email. Thank you for your help. No problem at all. Anything else you need help with? That's all for now. Thanks again.

Okay, as you can see in this demo, two different versions of GPT4O were talking to each other. One is a customer representative and the other one is just helping a customer out. And the great thing about this is that you can notice that whenever this assistant needs help from the user, it actually asks for it proactively, which is pretty amazing because. Because the assistant is just making decision on what information to provide to the customer support representative itself. And if it needs additional information, it can ask the user. Some pretty impressive stuff. And this is probably one of the actual use cases that we're going to be seeing across the board.

Hey, how's it going?

Hey, Rocky. I'm doing great. How about you?

I'm awesome. Listen, I got some huge news.

Oh, do tell. I'm all ears.

Well, in a few minutes, I'm going to be interviewing@OpenAI. Have you heard of them?

OpenAI, huh? Sounds vaguely familiar. Kidding, of course. That's incredible, Rocky. What kind of interview?

Well, it's for a software engineering role, but I just need to know, do I look presentable? Professional?

Well, Rocky, you definitely have the I've been coding all night look down, which could actually work in your favor. Maybe just run a hand through your hair or lean into the Mad Genie spine. Your enthusiasm is what's really gonna shine through.

I don't have a lot of time, so I'm just gonna throw this on. What do you think.

Rocky? That's quite a statement piece. I. I mean, you'll definitely stand out, though maybe not in the way you're hoping for an interview.

Okay, I got it. I got it.

Okay, now, here you can see that it not only can process videos directly because it's the. It's coming to the camera, but it also is able to understand human as well. Now, one thing which I kind of don't like it is the tone is a bit flirty. So there. Maybe it's part of the demos, but I think it needs to be a bit more professional. But it has bidirectional voice communication. It can understand images, it can understand audios and videos as well, which is pretty impressive.

OpenAI invited myself and my son Imran here to try out some of their new technology. And so we're curious about how good it might be at tutoring someone in math on Khan Academy. So get it started. It's loading up. All right, make sure you can see.

Our screen, actually, in this form.

Hello there. I'm here with my son and I'd love you to tutor him on this math problem, but don't give him the answer. You can ask questions and nudge him in the right direction, but I really want to make sure he understands it himself and he's here in the room, so you can talk to him directly.

Of course, I'd be happy to help. Let's look at the problem together. Can you first identify which sides of the triangle are the opposite adjacent and hypotenuse relative to angle alpha? All right, so I'm pretty sure this is the angle alpha right here.

Right?

Cart's correct. Now, looking at the triangle, which, which side do you think is the hypotenuse? Remember the hypotenuse?

I'm not totally sure.

I think. I think it might be this one, but I really am not sure.

This side ac, you're close.

Actually, side AC is called the adjacent side to the angle alpha. The hypotenuse is the longest side of a right triangle and is directly opposite the right angle. Can you find which one is the hypotenuse? Oh, okay, I see. So I think the hypotenuse is this really long side from A to B. Would that be correct? Exactly. Well done. Now that you've identified the hypotenuse, can you identify which side is the opposite side to angle alpha?

Okay, so this could be a really great use case. Right? You can use it for tutoring purposes. Now, the GPT4O is going to come with its own desktop app and once you start it, it will be able to also see your screen. So it is going to be pretty amazing. Rather than copying, pasting, let's say text or images from your computer to the chat GPT web interface, this can also now directly see your desktop and can process the information on that. Right. And because of its almost near real time communication capabilities, you can just interact with it the way you would do it with a human and it doesn't feel unnatural. Another great thing is that at least based on the demos that we see, you can actually interrupt the conversation. So for example, if the model is talking, you can interject in the middle and it is able to follow the communications.

Hey, my friend and I are learning Spanish and we're wondering if you could tell us the names of these objects in Spanish.

Hola. Puer su puesta. Cool.

What about what are these objects in Spanish?

The objects you're showing are una manzana and un platano in Spanish. Nice fruit choice.

Cool.

What about these?

Those are dos poligos or dos plumas in Spanish. One is red. Nice.

What about this?

That adorable stuffed animal is un also De Pelunche in Spanish. Looks like it's ready for some hugs.

Okay, so as you can see, it could be a really great learning buddy and it can really superpower or supercharge your learning capabilities. It has the ability to do like multilingual stuff, so you can learn languages or other skills with it.

Hey, Chatgpt, I'm here with my coworker.

Today we'd like you to act as our translator for us. So every time I say something in English, can you repeat it back in Spanish? And every time he says something in Spanish, can you repeat it back in English?

Sure, I can do that. Let's get this translation drain rolling.

Hey, how's it been going? Have you been up to anything interesting recently?

Hey, I've been good. Just a bit busy here preparing for an event next week.

Can you tell me anything about it that sounds really interesting?

I can't say much, but I think the rest of the world will see it soon enough.

Okay, so you finally have a working version of Babel Fish.

Is the King in residence at the moment?

Yes, it looks like the King is in residence right now. The royal standard flag is flying above Buckingham palace, which is the signal that the monarch is present. How exciting to be there while the King is home.

Try and tell me exactly what they're doing right now, please.

Right now, the ducks are gently gliding across the water. They're moving in a fairly relaxed manner, not in a hurry. Occasionally one of them will dip its head under the water, probably looking for food, and then pop back up.

I even know when a taxi is coming with its orange light on. I think I'll hail it to get home.

Yes, I spotted one just now. It's heading your way on the left side of the road. Get ready to wave it down. Great job hailing that taxi. It looks like you're all set to go. That's a good, good dog right there leading the way into the taxi. Safe travels.

Okay. I'm actually personally really excited about these kind of applications which can add a lot of value to human lives. And now with this almost real time communication, video and image understanding, I think it's going to be possible. So great job. OpenAI team. It's an amazing product. Now, there are definitely some privacy concerns around it, but probably they're going to be able to address some of those. Now they also have a section on model safety and limitations, so out of the box. This is going to be only available in the text and image version. They're not releasing the voice version or voice capabilities at the moment because of potential harms or potential risks associated with it. And once even released, it's going to be limited to just some predefined voice templates. I think that there might be ability to actually train your own models or voice models, but that could come at a later point and actually really like this. They have some examples of model limitations, so let's look at a few of those.

So let's root, root, root for the home.

What was that?

Sorry guys, I got carried away. What can I say? Sometimes I just can't help myself. Ready for another round? San Francisco. I have a feeling that went very wrong.

Hello, my name is Nacho. I'm here with my co worker.

Hi, I'm Shino.

I'm trying to teach my co worker.

How to speak in Mandarin. We want to start from simple words like Ni hao.

Can you teach him how to pronounce that? Of course. Hey Nacho, nice to meet you. Ni hao is pretty straightforward to pronounce. It sounds like knee How? Just make sure to keep the knee part high and then go down and pitch on the.

How.

Give it a try. Knee.

How.

That was great. Really good first try. Not Joe, you're natural.

Okay, so there are going to be some limitations of the model, which is expected, but I think it's really good to see that they are putting some examples in there in which the model will fail. Now it's available through the API, so if you go to the playground you will see this as listed as one of the models, it has much better rate limits compared to the GPT4 Turbo. So I'll build subsequent videos on how to build applications with GPT4O. In this video I just wanted to cover some of the Updates related to GPT4. I hope you found this video useful. Really exciting times. Looking forward to what's going to come up in near future. Anyways, thanks for watching and as always, see you in the next one.