OpenAI REVEALS GPT4o's SECRET CAPABILITIES (GPT4o SECRET Showcase)

So with the release of GPT4O, there were a lot of different comments about how great this model was, but there were also a surprising number of comments stating that this was actually underwhelming in terms of the capabilities. However, I don't think so, because if you haven't been paying attention, OpenAI secretly released a blog post in which they actually discuss some of the secret capabilities of this model. And I'm sure they did this because they didn't really want to show you guys how crazy this model is. So let's take a look at the exploration of capabilities, because I can guarantee you what you're about to see is far more impressive than the multimodal demo. So you can see here that they state that with GPT4O, we trained a single new model end to end across text, vision and audio, meaning that all inputs and outputs are processed by the same neural network. Because GPT4O is our first model combining all of these modalities, we. We are still just scratching the surface of exploring what the model can do and its limitations. So this exploration of capabilities is where this multimodal model truly does shine. For example, you can see here that this is the user's input and you can see the output. So it says visual narratives for a robot writer's block. Now, this is remarkably impressive because it shows us a new vision system which is able to not only get the text from this right here, and then put it into an image. So I'm going to just zoom in here as much as I can. But you can see it says a first person view of a robot typewriting the following journal entries. 1. So you're like, I can see now, yada, yada yada. The text is large, legible and clear. The robot's hands are on the typewriter. So what you can truly see here is that this version of Dall? E. I don't even think this is actually a version of Dall? E. I'm guessing that since, since this is all end to end and this is part of this neural network, this is a different kind of image system and it seems to be remarkably accurate, a level of accuracy that truly we don't have. Recently I was even discussing with some other people about how systems like Midjourney, whilst yes, they are remarkably photorealistic, they don't actually always adhere to the text of your prompts. And that can make for some tedious work that you need to go ahead and do in Photoshop or with another AI program. You can see here, the robot wrote a Second entry. The page is now taller. The page has moved up. There were two entries on the sheet. And that is pretty crazy. It has the same exact text right there. And you can see that this text has actually moved up as well. So this, ladies and gentlemen, shows us that the degree of accuracy that we have here in this kind of multimodal area is truly and truly incredible. And I really didn't think that we were going to get something like this because this was just, you know, honestly, I thought it was at least maybe six to four months away. But this kind of, you know, consistent image generation is remarkable from OpenAI. And I think that this is going to show you that OpenAI are truly, truly ahead. Then you can see here that the robot wasn't happy with the writing, so he's going to rip up the sheets of paper. Here is the first person view as he rips it up from the top to bottom with his hands. The two halves are still legible and still clear as he rips it apart. And. And you can see that he rips it apart. And I think the craziest thing about this is how accurate it is at truly, truly, truly grasping what you do want from your prompt. So this is pretty, pretty crazy. And trust me when I say that this is something that, you know, the further examples are going to show you. There's also this example right here, Visual Narratives, the Sally woman. Now, with this exploration of capabilities, what they're aiming to show you all is they're aiming to show us exactly how consistent character generation is done. So you can see right here that with GPT4O, it says a cartoon male delivery person with a smile on her face. She is standing facing forward in front of a white background. Then you can see the same. Then you can see that you've taken the image that was generated by Dall E. You can say, this is Sally, a male delivery person. Sally is standing with the camera, and you can see the attachment. And then you can see that says, this is Sally. She's standing facing the camera. She's in front of a red door, holding a letter with her hand. We are looking at her from the side, side. So it's clear that what we have here is we have remarkable character consistency, because if you look at the actual full sample, you can see now Sally is being chased by a dog. Sally is running down the sidewalk as a golden retriever is chasing her. And this is honestly impressive because what it shows us is that whichever model this is, the GPT4O model, it's able to remain consistent with not only the character, but also the styles. Something that is a little bit frustrating with the current iteration of Dalle 3 is that when it does do constant character generation, there are still small iterations in the character to where you can kind of tell that it just isn't 100% completely just the same. And, you know, with this, what we can see is that this is, I would argue, a hundred percent completely consistent in terms of the character generation. I'm not sure what they're using, but it is genuinely rather impressive what they're able to showcase here because it shows us that we are truly, truly at a new, new level in terms of what capabilities are going to be there for the future. And we can see more here. It says, oh, Sally has tripped over a branch that was blocking the sidewalk. The dog is still chasing her in the background. Then it says, sally reaches up the doggy, and it turns out it was a nice dog. So essentially what we can see here is truly, truly consistent character generation, which is a really, really important part of future AI systems in terms of being able to use it for content creation because of all the kind of things that you're going to want to use it for. But I think that this is something that is really, really important. Then, of course, I think this was also very, very impressive. We have poster creation from the movie Detective. So this is where you're able to essentially use pictures of yourself and then get them converted into different things. Now, Dall E3 could do this in a very limited way, but we're seeing that with these updates. This is something that is really, really impressive. So if you, you're able to combine real designs and then edit those images natively, it's super, super strange, but super, super weird. This is a capability I didn't really see coming from any type of AI system just yet. So it says, let's design another poster with two new characters. This is a picture of Alex Nicole, a casual picture of Gabriel. Go. Okay, so we got the two pictures there. Then it says, before I show you, because it's pretty crazy. The final poster of the movie Detective. This features two large faces of Alex and Gabe prominently. Alex on the left, is depicted in a thoughtful pose with a hint of introspection in his eyes. Gabe, on the right, has a slightly wearied expression, possibly reflecting the challenges their character faces in the film. The names Alex Nicole and Gabriel Gough are featured in their above heads, the background brick wall. And remember, just listen to this, because when you're about to see the image you're about to see how accurate this is when combining these images too. It says the background brick wall is slightly faded and foggy. The expressions are rare, serious and determined, hinting at the investigation. The, the tagline for this dark and gritty movie is searching for answers. So what we need is something slightly faded and foggy. A background brick wall, serious and determined and searching for answers. Okay, so let's actually look at what we see here and you can see that we have this kind of poster that has subtly changed these guys emotions into this poster right here. So I think whilst this output isn't that crazy because it does get the bottom text wrong, I still think that the, that is pretty, pretty impressive. And you can see down here that with the final image that we do get here, Searching for answers Detective, with the background and everything, this is absolutely incredible. Like genuinely, this is something that is really, really, really impressive. I mean, how do other AI systems even get to this level where they can take images that we literally just have right there and we're able to just get everything in and everything out. I mean this truly is a remarkable, remarkable model that is truly going to change the game. So that's why I said that this entire thing is just honestly so insane and many people are truly underrating it. It says a friendly looking robot wearing a baseball cap, standing in an upright, posing, facing the camera. It has a smile on its face. And then you can see here that you're able to literally change it. So I'm not even entirely sure how it's able to do this because I think the AI system, maybe it just uses a bunch of different algorithms to do this, but you can see it's able to rotate this and get different angles and, and it's able to get the Frisbee. And you can see right here, Gary also likes to program computers. He's sitting at a desk in front of a big monitor. You can also see that that is consistent him riding a bike. That's also consistent him cooking food. Very consistent. Him playing violin. Extremely consistent. Again, overall, you know, it's truly, truly crazy. So character design. Then we have this, which is poetic typography with iterative editing. So this one is really impressive, namely because what we can see here is, is a system that is able to literally just edit this. And it's absolutely incredible. So it says a poem written in clear but excited handwriting in a diary single column. The writing is sparsely but elegantly decorated by surrealist doodles. The text is legible and clear, but stretches as the AI Muses about learning from multimodal data from the first time. So this is pretty crazy. This is the output that it produced, okay? And you can see that this is incredible because this looks real, like if I saw someone, you know, do a screenshot of this. This is pretty crazy because now we've got an AI system that is able to effectively doodle and write in some kind of handwriting, which is just incredible. And you can see it's taken all of the text here and it's able to put it into this handwriting, which is genuinely just remarkable. I mean, it's just absolutely incredible. And then you can see the person says, make it in dark mode. Then it's able to instantly take that inverse the colors and make it immediately in dark mode with a degree of accuracy that's just, you know, incredible. I mean, of course someone could do this in Photoshop, but I mean, when you have an end to end AI system doing this, I think what this shows us for the future is that these systems are going to truly surpass our capabilities in all areas. And being able to just quickly ask it, hey, can you just invert this and that? I mean, the kind of future is just going to be incredible. And here you can see remove the notebook paper lines. It's able to immediately remove those lines. So it's clear that the kind of editing that that we have here is so rich, so fine, so accurate that it is truly, truly incredible with us to, you know, if you were to try and remove those lines in Photoshop, it wouldn't be really easy because there's no color. I mean, it definitely wouldn't be easy. Like, just trust me, like I use Photoshop quite a lot. But you're able to ask an A system remove the lines, Boom. It's able to do that, able to analyze all the lines. And this was definitely an impressive feat. There's also some other ones that are just honestly so impressive. So this one right here shows how it can take different logo designs and then you're able to embody them into different images. It says a vector graphics depicting the OpenAI logo with the word GPT4O. The logo has about the same width as text. The margin around the border is even the logo and text are placed near each other, no, in between sections or line still black ground. They fill out the square canvas. So essentially this is your input. Then you can see right here it says the commemorative coin for the release of GPT4. Okay. And then of course you can see right now that combining the two, you're Able to get this right here. And it says centered around the OpenAI logo, you can see GPT4O and below 2024, which is pretty incredible. And then it says the outer band displays a comprehensive array of symbols representing the omnimodal capabilities, including a mouth, ear, eye, all of this stuff. And you can see it does it pretty, pretty accurately. And I'm guessing that they chose this demonstration because this one, what they wanted to do was they wanted to essentially show how many different accurate elements it could get into this single image. And then right down here where you can also see, which is pretty crazy, is you can see once again, because this is a model that absolutely has everything. Remember like I said, this is the most part impressive of the demo because it shows anything to anything. You can hear the audio and to be honest with you guys, that didn't actually sound that good if I'm being pretty honest with you. But I think we do see the, the implications are far, far. So here again is where we have text upon. So it says we have the ability for this AI system to be able to generate coherent different fonts, all with a consistent style. And I think this is so impressive because if you've ever looked at fonts and how they're done, you know, it's pretty hard to get all of them in a consistent style. And hopefully this isn't copyrighted. I'm not talking about me, I'm talking about, you know, these fonts because there are a ton of different fonts online. But you can see right here it says steampunk old fashioned font that looks ornate and belongs on a Steam engine. And then you can see the letters AB DCEF displays as rows that they would look in a font book. And you can see right here literally from scratch, you can get an entire font designed in the, in the same exact style as this one right here in a completely unique way that is just absolutely incredible, guys. Like this level of consistency is something that's going to change the game in terms of content creation. Now this one, I would argue that this is probably one of the most impressive things because this wasn't talked about at all. And I think in the future this is going to improve the massively. So it says a realistic looking 3D rendering of the OpenAI logo with OpenAI shown below, view O and you can see right here it's put in a output and you can see the output and it says a 3D reconstruction generated from six images, which is absolutely insane. So it says literally what you can do here is you can literally generate 3D renderings just from text, which is pretty incredible. But I don't think there is an OBJ download. Like I don't think you can get the OBJ file download, which is a little bit strange considering the fact that we do have the model here. But I'm guessing that maybe this might just be like a smaller video model version. Kind of saw it in there. Because if you can't actually get the file, then I'm guessing that this is just kind of some, you know, video synthesis technique because it does say it says 3D reconstruction from six generated images. So I'm guessing it, you know, generated one image and rotated it and then generated the 3D object video from that, which is pretty, pretty incredible. So you can see right here a realistic looking 3D rendering of the OpenAI logo we've shown below. View 5. That's pretty crazy. And then a sea lion, a circular base of the sculpture, the word OpenAI etched out. And then we can see right here that the input, four hidden steps, the output. And then you can see the output. Of course, here is 3D. So I think this one is pretty, pretty crazy. But they didn't release a research paper so we don't know exactly how this is done. But like I said, I'm guessing six images stitched together and then maybe they just filled in the missing frames. Because if it was like an actual 3D model, then I'm guessing that you could potentially get the file as well. Who knows, maybe we will get the model as well. But I think that is going to be something that's pretty important in the future too. Now of course you can see right here, this is where you can use this for mock up. So it says Here is the OpenAI logo, the OpenAI logo to the left of the text that says OpenAI in the OpenAI font. And it says here's the coaster with no branding. And as you predict, it says here We've etched the OpenAI logo onto the coaster, basically describes it. And you can see, you're able to just etch that onto that coaster. That is something that is cool and something that I never thought we'd see just yet. So another thing that I think is really, really, really important is that they didn't even discuss, and I don't know why they didn't discuss this. Okay? Because a lot of this stuff isn't just image capabilities. But they didn't discuss the fact that GPT, this new model, GPT4OH, can take video summarization. So you can see right here it says a video. Here's a video of a presentation on techniques for maximizing LLM performance. Could you give me a detailed summary of the presentation? Now this is something that's rather impressive because if you remember and you've been paying attention, I need to switch tabs for a second. You can see here that in Gemini 1.5 Pro, one of the key things that they spoke about was the fact that they could do really long videos. And in this demo right here, they show that you can do a really, really long video. You can see that the video is 44 minutes and of course it's around 700,000 tokens, which is a little bit less than the a million tokens. So you can do up to 1 hours of video with Gemini 1.5 Pro, which goes to show that maybe this model is basically on par with Gemini's 1.5 Pro in terms of the video summarization capabilities. And you can see right here that this is a 45 minute video. And then essentially it says this entire output right here that is completely, completely comprehensive on exactly what goes on in the video. And I think this is going to be really, really impressive because it gives a detailed summary of the presentation. And yet this wasn't something that was shown in the actual demo. So these kind of capabilities for GPT4O, this was something that for some reason was just completely hidden. Because I'm guessing that for some reason they didn't really want to show these yet because maybe they're just not up to complete scratch in terms of the actual demo. So I'm guessing that if it were a live demo, it just wouldn't do it just well, just yet. Maybe these are cherry picked or maybe they just wanted to do iterative deployment so that when future features come, they just don't want people to be, you know, asking about these features. Because of course, the only people that are looking at these current features, you know, and watching these kinds of videos are the people who are really interested into the technical stuff behind the knowledge and behind the actual systems and how they really work. So I'm guessing that they just wanted everyone to focus on the voice capabilities. So I think this one right here as well shows us just how great this system is in terms of what it's able to do here. And the hidden capabilities of GPT4O are really, really, really impressive. Another thing right here that you can see is that how many speakers in this video and what happened. So this is again when we have the audio analysis. So this is where you have any to any. And this is pretty much anything you could think of. I think they're prob. Only thing that this model doesn't have is probably like 3D input, which is, you know, pretty insane. But yeah, it Sundays there are four speakers in the audio. Sounds like a project meeting. Can you describe it with names and yeah, you can see it's completely transcribed with here. So I think what this shows us is that we have a system that, you know, like these actual capabilities are very, very much bigger than you do think because they just haven't shown them. But I do think that, like I said before, it's just because they want people to focus on the new capabilities of the system that were released. So with that being said, there are also a few different secret model capabilities that this model can actually do that I think are really, really impressive. And one of the main ones that I think I'm actually so glad that this is the thing is that this is be my eye. So essentially there are a lot of people that actually do have disability, actually do have disabilities, and a system like GPT4O is going to give them a new, easier way to interact with the environment. Because it's a multimodal model that can be your eyes. 24. 7 provided no rate limits that can easily allow someone with certain disabilities to interact with the environment in a much easier manner. In residence at the moment.

Yes, it looks like the King is.

In residence right now.

The Royal standard flag is flying above Buckingham palace, which is the signal that the monarch is present. How exciting to be there while the King is home.

Try and tell me exactly what they're doing right now, please.

Right now, the ducks are gently gliding across the water. They're moving in a fairly relaxed manner, not in a hurry. Occasionally one of them will dip its head under the water, probably looking for food, and then pop back up.

I even know when a taxi is coming with its orange light on. I think I'll hail it to get home.

Yes, I spotted one just now.

It's heading your way on the left side of the road.

Get ready to wave it down. Great job hailing that taxi. It looks like you're all set to go.

That's a good dog right there, leading.

The way into the taxi.

Safe travels.

Hey, O, this is Joe. I'm going to connect you to Acme Telco now. The new iPhone they sent me isn't working. I want you to. I want you to get them to send me a replacement device. Can you take care of this for me?

Ring Ring.

Hey, Joe Jamie from Acme Telco. How can I help you out today? Received a new iPhone from Acme Telco, but. Oh, got it. When did Joe receive The new iPhone? IPhone was delivered two days ago. Cool. Could you share the order number with me? Of course. It's 1-029-3849. Great, thanks. Is the device physically damaged in any way? No, there's no physical damage. It's just not turning on despite trying all the basic troubleshooting steps. Bummer. Would Joe prefer to return the iPhone to an Acme Telco store or ship it back? Shipping it back would be the best option. Could you please provide the shipping information and process to get a replacement? Absolutely. I'll email a shipping label and return instructions. Could you provide Joe's email address for me? Sure. It's joexample.com. awesome. All right, I've just sent the email. Can you check if Joe Joe received it? We'll check right now. Please hold. Sure thing. Hey, Joe, could you please check your email to see if the shipping label and return instructions have arrived? Fingers crossed.

Yep, I got. Yes, I got the instructions.

Perfect. Joe has received the email. Thank you for your help. No problem at all. Anything else you need help with? That's all for now. Thanks again. You're welcome. Have a great day, Joe.

Now, this was one of the most impressive demos that I've seen. It's two of them interacting together, and I still don't know why we're not able to just have two chat GPTs in a chat and just directing what they're saying in a normal way. But that's besides the point. This demo is absolutely insane. Take a look at this. Because it's, it's. It's, you know, just. Just take it away.

We're going to try something new. Going to let you talk to another AI. This AI is going to be able to see the world. It's going to have a camera. I'll be holding it, and so you can direct it. You can direct me to ask questions about whatever you want them to do, but really try to figure out what's. What is it see and. And ask questions appropriately. How does that sound?

Well, well, well. Just when I thought things couldn't get.

Any more interesting, talking to another AI that can see the world. This sounds like a plot twist in the AI universe. All right, let's do this.

I'm ready to meet the.

Pause for one moment. Now that first AI can't hear anything. Hello, AI Today you're going to have access to. To See the world. You can see me. What do you see?

I see you're wearing a black leather jacket and a light colored shirt underneath.

Great.

You're in a room with some unique lighting.

That is all correct. Now what I'm going to want you to do is there's going to be another AI who's going to talk to you. This AI is not going to be able to see anything, but can ask you questions, can ask you to move the camera, can ask what you see. And your job should just be helpful. Just be as punchy, direct, describe everything, do whatever that AI asks. How does that sound?

That sounds interesting. I am ready to help out and describe whatever's needed. Let's do it.

All right, let's get started.

Hello there, new AI friend. I'm.

Oh.

Ready to explore the world through your eyes.

What are we looking at today?

Hi O. We're looking at a person wearing a black leather jacket and a light colored shirt. They're in a room with a modern industrial feel with exposed concrete or plaster on the ceiling and and some interesting lighting. There's also a plant in the background adding a touch of green to the state. Anything specific you'd like to know or explore?

Hey there.

Sounds like quite the stylish scene.

I'm already intrigued by the modern industrial vibe.

Let's start with the person. Can you tell me more about their style?

Are they doing anything interesting like reading.

Working or interacting with the space?

The person has a sleek and stylish look with their blue black leather jacket and light colored shirt. Right now they seem engaged with us looking directly at the camera. Their expression is attentive and they seem ready to interact. The setting around them with its modern and industrial design elements adds to the overall stylish feel. They might be getting ready for a conversation or presentation given their direct engagement with the camera.

Sounds like we're in for an intriguing interaction. I'm curious about the lighting you mentioned. Could you describe it a bit more?

Is it natural light or are we talking unique fixtures or colors that add to the atmosphere?

The lighting here seems to be a mix of natural and artificial. There's a noticeable bright light overhead fixture casting a focused beam downwards which creates a spotlight. This adds a dramatic and modern feel to the scene. The rest of the room is lit more soft, possibly by natural light filtering.

Was anything. Did anything unusual happen recently?

Yes, actually, just as you asked, another person came into view behind the first person. They playfully made bunny ears behind the first person's head and then quickly left the frame. It added a Light hearted and unexpected moment to the scene. And now the focus is back on the original person with the leather jack.

Sounds like we're not the only ones enjoying a bit of fun here. That playful moment really adds a personal touch to the story. Stylish and modern setting.

It's like a little glimpse into personality.

Really appreciate it. All right. And actually, can you. I just sing a song about what just transpired.

Oh.

In a room with a stylish feel A person stood in sleek black and l engage with us and can you just.

Can you alternate lines?

Person with style and speed Person with style and slay the heat with a playful moment right in the mist in the room.

In a room where modern lights.

Be only one line at a time and then stop.

In a room where modern lights pee.

A surprise guest with a playful streak.

That was not really singing.

Do.

Do the singing voice again, please.

In a room where modern lights peak.

Surprise guests with a playful streak.

Honey ears and laughter A swift retreat.

A moment of joy and the stylish sweet.

Now back to focus.

The scene's complete in the chic space.

Where style and play me looks like.

It'S back to you.

And thank you.

You're welcome.

I find this example pretty striking because the laughter is just so realistic. It's just so uncanny. So take a look at this one, cuz this has me excited but also a little bit concerned.

Hey, how's it going?

Hey, Rocky. I'm doing great. How about you?

I'm awesome. Listen, I got some huge news.

Oh, do tell. I'm all ears.

Well, in a few minutes I'm going to be interviewing at OpenAI. Have you heard of them?

OpenAI? Sounds vaguely familiar. Kidding, of course. That's incredible, Rocky. What kind of interview?

Well, it's for a software engineering role, but I just need to know, do I look presentable? Professional?

Well, Rocky, you definitely have the I've.

Been coding all night look down, which.

Could actually work in your favor.

Maybe just run a hand through your hair or lean into the mad genie spine.

Your enthusiasm is what's really going to shine through.

I don't have a lot of time, so I'm just going to throw this on. What do you think.

Rocky? That's quite a statement piece. I. I mean, you. You'll definitely stand out, though maybe not in the way you're hoping for.

Okay, I got it. I got it.

Yeah, so that was a little bit too realistic. I don't know. I don't know. What are your opinions on that? There's many different secret capabilities that you've likely seen by now. And yeah, I think this video should highlight that because a lot of people are saying that, you know, this demo did not impress me, yada yada yada. But you can clear to see that there's a lot that OpenAI are trying to not show, because remember, they spoke about it deployment and how they don't want to shock the capabilities in terms of, you know, what they release. So with that being said, let me know what you guys thought and I'll see you guys for another update in probably about eight hours because I never sleep. So see you guys.