OpenAI's NEW AI "SORA" Just SHOCKED EVERYONE! (Text To Video)

What you're looking at is 100% AI generated. This is OpenAI's state of the art new text to video model. They state Introducing sora, our text to video model SORA can generate videos up to a minute long while maintaining visual quality and adherence to the user's prompt. Today, SORA is becoming available to red teamers to assess critical areas for harms or risks and and we are also granting access to a number of visual artists, designers and filmmakers to gain feedback on how to advance the model to be most helpful for our creative professionals. They also state that we are sharing our research progress early to start working with and getting feedback from people outside of OpenAI and to give the public a sense of what AI capabilities are on the horizon. SORA is able to generate complex scenes with multiple characters, such as specific types of motion and accurate details of the subject and background. The model understands not only what the user has asked for in the prompt, which you can see in the bottom part of the video, but also how those things exist in the physical world. The model has a deep understanding of language, enabling it to accurately interpret prompts and generate compelling characters that express vibrant emotions. SORA can create multiple shots within a single generated video that can accurately persist in characters and a visual style. In research techniques, the SORA is a diffusion model which generates video by starting off with one that looks like static noise and gradually transforming it by removing the noise over many steps. SORA is capable of generating entire videos all at once, or extending generated videos to make them longer. By giving the model foresight of many frames at a time, we've solved a challenging problem of making sure a a subject stays the same even when it goes out of view temporarily. Similar to GPT models, SORA uses a transformer architecture, unlocking superior scaling performance, and they also state that we represent videos and images as collections of smaller units of data called patches, each of which is akin to a token in GPT. By unifying how we represent data, we can train diffusion transformers on a wide range of visual data than was possible before, spanning different durations, resolutions and aspect ratios. SORA builds on past research in Dall E and GPT models, and it uses the recaptioning technique from Dall E3, which involves generating highly descriptive captions for the visual training data. As a result, the model is able to follow the user's text instructions in the generated video more faithfully, and in addition to being able to generate a video solely from text instructions, the model is able to take an existing step, still image and generate A video from it animating the image's contents with accuracy and attention to small detail. The model can also take an existing video and extend it or fill in missing frames. And you can learn more in their technical paper coming today, which is likely going to be linked in the description. SORA serves as a foundation for models that can understand and simulate the real world capability and we believe will be an important milestone. So ladies and gentlemen, will we have it here? This is by far the most shocking thing I've seen in AI. After we got to see the GPT4 capabilities reveal, this text to video model is genuinely absolutely incredible. Based on what we are seeing here. I still to myself cannot believe that this clip, the one that I'm looking at right now, is actually AI generated. I mean it looks like a realistic clip first of all. And not only that, it is also so high quality that it makes me think that this is even photorealistic. I mean, take a look at this example right here. Looking at this, if someone was to show me this video perhaps a couple of months ago or even a couple of days ago, I would say with striking accuracy that yes, I can approve that this is 100% realistic. But now we really, really don't know what is going on because we this is the next evolution in text to video. There have been a ton of different competitors and companies racing towards the giant title of the state of the art model, but OpenAI has shown us what is capable. Now there are so many different examples that OpenAI have included in this blog post, but I'm going to be showing you guys some of my favorite ones that show you why this is a truly advanced model that really does understand exactly what's going on. So you can see this example of a stop motion animation of a flower growing out of the windowsill of a suburban house. And the reason I like this one so much is because this shows us how well the model understands exactly what is going to come next and true coherence in terms of what to expect from the final output. Because it's not like previous models in which we just get a literal text to video. We can clearly understand that this AI system has some general world model. And that essentially means that this model has a true understanding of physics and how things move and how they exist in the world. Now in addition, they also did state that there are several model weaknesses. So they state right here that the current model has weaknesses. It may struggle with accurately simulating the physics of a complex scene and and it may not understand specific instances of Cause and effect. For example, a person might take a bite out of a cookie, but afterwards the cookie might not have a bite mark. The model also might confuse spatial details of a prompt, for example, mixing up left and right. And may struggle with precise descriptions of events that take place over time like following a specific camera trajectory. Let's take a look at some of these weaknesses. So we can see here that this is one of the first demos of the model's weaknesses. The prompt is step printing scene of a person running cinematic film shot in 35 millimeters. And to the normal eye. Of course this does look pretty realistic just to state that out there like it does. But of course the weaknesses is that they state that SORA sometimes creates physically implausible motion. For example, if you haven't realized the person is running backwards on a treadmill when treadmills don't really do that. I mean, I guess you could kind of run like this if you made your own treadmill. But. But this is not generally what we expect when we see someone looking and running on a treadmill. Now there was also this example, the first one. And although they state that this is a weakness, I still find this video to be really cool and one of my favorite because it is basketball through hoop and then explodes. And we can see that of course the basketball does go through the hoop and it does seem like a kind of slow motion video. It says the weakness there is that it's an example of inaccurate physical modeling and. And unnatural object morphing which is of course how the basketball goes through the rim right there and doesn't exactly, I guess you could say interact physically with that part of the rim. Now in addition here there was also this really cool example and it's prompted with archaeologists discover a generic plastic chair in the desert. Excavating and dusting it with great care. And of course the find really funny. And you can see that the prompt is right rather large. I mean it says. A grandmother with neatly combed gray hair stands behind a colorful birthday cake with numerous candles at a wood dining room table. The expression is one of pure joy and happiness. With a happy glow in her eye, she leans forward and blows out the candles with a gentle puff. And the cake has a pink frosting and sprinkles and the candles cease to flicker. The grandmother wears a light blue blouse adorned with floral patterns and. And several happy friends and family are sitting at the table can be seen celebrating out of focus. The scene is beautifully captured, cinematic showing a three quarter view of the grandma and the dining room. And warm color tones and soft lighting enhance the model. Now, essentially, the weakness here is pretty evident. At first, it isn't really evident because this looks really, really good. But if we take a look at this, what they state, they state that the weakness here is simulating complex interactions with between objects and multiple characters is often challenging for the model, sometimes resulting in humorous generations. So if you do take a look at the back, you can see that sometimes the woman is clapping. Like, if we go to the start, you can see that she's clapping and it looks really good. And then of course, over time, she's doing something really weird with her hands. And of course, the other person is kind of like waving in this sort of weird way with her hands, but at least they do have five hands, so it's not that crazy. And of course, her eyes just seem to be looking around in a strange way. But I still think that this is this bad clip from Opry is still better than some of the other clips that exist from state of the art models from other companies. And of course, and the reason I love this clip so much is because if we take a look at the Dalmatian trying to move over here, it's crazy because it clearly understands how the dog's legs move as it jumps from one ledge to another. So this is some really, really, really advanced technology that exists here. And what's crazy about this is that I recently just saw a tweet of Sam Altman actually showcasing exactly what this model is like. So what I'm going to show you guys is a few of the tweets that Sam Altman has directly replied to to showcase how crazy this model really is. And there is a lot to unpack here. So let's get into some of these examples. So one of the ones here is a prompt from a random person. That's why this is so cool, because it's interactive. A wizard wearing a pointed hat and a blue robe with white stars casting a spell that shoots lightning from his hand, holding an old tome in his other hand. And we can see at full screen this looks really, really good. It seems to be pretty, pretty cool. And at the end, you can see he actually cast the spell and there's some kind of orange thing going on there, which is rather fascinating because we really want to understand how these models are able to do what they do. There was also this one. A half dark, half dragon flies through a beautiful sunset with a hamster dressed in adventure gear on its back. And this one is rather accurate, and it looks like some kind of 3D animation from a kids show. But if we put it to full screen, you can see that they clearly do have access to this model in terms of it being really responsive and really rapid in terms of its development. And the reason I do like the fact that Sam Altman is going to give these tweets a response with the actual video output is because he's showcasing that the model that they've built is actually a really good model and they haven't cherry picked any of the videos that have been outputted throughout from their system. So him going through all these tweets and responding to people and showcasing what the model capabilities are shows us that this is likely a very capable text video model. Now, this one is one of my favorite ones. A street level tour through a futuristic city in which harmony with nature and simultaneously cyberpunk high tech. And this looks pretty crazy. If I saw this as the background of something, I would say that there's no way that this isn't CGI or some kind of thing 3D render that someone did in some kind of 3D application software. But of course you can see that this is literally text to video. And by far this looks so, so realistic in terms of how it was created. I mean, the humans walking around, it's just absolutely incredible. And that was literally someone that just did that prompt to Sam Altman. And what was funny was with the recent prompt that we just got, we got an updated one from Sam Altman where he said, look even a better one. So he prompted it again to see what the model would do. And you can see that it looks like a hamster with adventure gear on. And it's chilling on this half a duck, half dragon type creature. There was also this futuristic drone race on a sunset in a place in Mars. And this is pretty crazy because this looks incredible. Like, I think the reason this is so interesting is because when we get an image from a text to image model, it's super interesting because we just get a snapshot of what could be some. Some kind of story. So I think the fascinating thing about these videos is that they do have sometimes their own storylines. Like, you can see that these drones are kind of racing around these kind of beams that seem to be the kind of guardrails for where they should be racing with. And I think these kind of mini stories that we do get are part of the interesting story and how we kind of understand how crazy these models are in terms of their capabilities. And this video right here is just going to show you how to how crazy OpenAI's image model is, because I'm pretty sure that whatever model they're using, they definitely do have an upgraded version of Dall E, which is purely focused on photorealism. And there were some people stating that, you know, OpenAI wants to use their photorealism for text to video. And it seems like that is really true because this prompt right here, this is just absolutely incredible because if someone was to show me this, I'd be like, why on earth did you put those two dogs there with a microphone and your headphones just for a picture? But it's a rather funny thing that you'll all enjoy. And if you take a look at this example that Sam Altman just tweeted out, this is crazy because we can see that the woman actually grabs a spoon and then starts moving some of the stuff in the bowl and then crazy enough picks up the bowl with that stuff in there. So the consistency of these videos is absolutely incredible. And it's nothing like what we've seen before in terms of how, how well the final output does look, because there is a clear understanding of physics and how things interact with one another. And there was this example here tweeted out by a researcher at OpenAI, Bill Pebbles. And you can see right here that he tweeted, welcome to Bling Zoo. This is a single video generated by Sora, shot changes. And all what's crazy about this model is that it's able not only to generate realistic text to video, but, but it's able to generate some kind of small movie. You can see here that all of these shots, the way that they're panning, the way that they've moved, the way that they've changed, are very, very cinematic. And this seems to be really, really, really advanced in terms of what we're seeing, because it's not like it's just generating these short clips, it's generating them and it's telling a story. So being able to generate this entire thing with shot changes is actually absolutely telling in terms of what this model is truly capable of. Now, there were some other videos that I did. Now there were some videos that I do want to show you. For example, this one right here, that is absolutely incredible that show us the kind of stories we do get when these things are left to run for longer durations. You can see that it's absolutely incredible in terms of the panning around and looking at the scenery. I mean, I'm honestly struggling to put words together to describe what I'm looking at. Because the fact that this is is AI generated truly, genuinely blows my mind. And I'm struggling to put words into what I'm looking at here. But all I can say is that the people that work at OpenAI are truly, truly ahead of where anyone really thought they were, especially with their video models. And there was also this example right here, prompt, of historical footage of California during the gold rush. And this looks absolutely incredible because with AI generated videos, a large problem that many have had is the fact that they're not high quality. So when there are essentially like in 1080p footage and you do kind of have small errors, you do really see them. But in these western type of old videos, the minor, minor mistakes that you do usually get, we're simply not seeing. Now, there's some other crazy examples that I do need to show you the capabilities. And by far, if there's one example that you do want to see, it's this one. Because this is reflections in the window of a train traveling through the Tokyo suburbs. I mean, ladies and gentlemen, you have to give OpenAI their credits. Because being able to make a video that can literally pan across right here, then when the reflection comes up, it's able to understand the lighting, able to understand how all of that works. I mean, generally, I have no words like OpenAI telling me that this video right here on my screen right now is AI generated. I genuinely don't believe that this is AI generated. I mean, I'm gonna have to wait until the model comes out itself because part of me wants to not believe it, because it just seems that my eyes cannot deceive me. And I truly cannot tell that this is AI generated. Even as someone who looks at tons and tons of different research papers and tons and tons of different AI videos, this one is by far the most realistic. I mean, some of the other ones, like this one, you could know, potentially tell that it's maybe 3D animated. And even this one looks 3D animated. It doesn't look AI generated at all. But, you know, some of them just don't make sense. And one of the capabilities that I wanted to show off from this model was this one. So this is a beautiful silhouette animation, shows a wolf howling at the moon, feeling lonely until it finds its pack. And you can see that there is a kind of animation there that this story goes on. So I think this kind of model that we do have is so advanced at generating, you know, single shot videos, like essentially kind of mini movies, and of course, panning around to showcase what else is in the environment and sticking to the entire storyline. Like the shot changes that you're seeing here are truly, truly incredible. I mean, at the end you're going to see that this wolf does manage to find its pack. And I wonder in the future if, if. Because we do have these things where you can have an entire storyline with this. Are we going to get music tracks with this? Are we going to get voiceovers with this? And are we going to get, you know, of course, ChatGPT and GPT4 creating entire movies that are just AI generated for us. Like imagine you're bored and you decide, you know what, I want a movie about this and about that. And then you go into an AI system and then you essentially say, I want a movie about this and about that and then output that entire movie for you. We could potentially be moving to an era in the future where we have 100% AI generated content that is literally user specific. I mean, of course that is a far, far away time. But the level of quality that I'm seeing in these clips is by far something that I thought we wouldn't see for maybe at least another year or another two years. And some people did make predictions that this was going to be the year of Hollywood level AI movies. And, and I do think that those predictions were really, really correct. I mean, take a look at this. It really does understand exactly how snow interacts with the fur. And you know what the craziest thing about all of this is? Is that sometimes, you know, when you've looked at the backend of CGI and visual effects studios on the amount of computations and calculations that they have to do for things like water and for things like fluid simulation. You guys have to understand that it takes hours, hours, even days, sometimes even months to render a few seconds of these advanced complications. So this is why this is so crazy, because their AI system is able to do that with a level of accuracy that we really didn't think was possible. Now I want to get into some of the comments because one of the comments from senior AI researcher at Nvidia also does touch on this. He stated, if you think OpenAI Sora is a creative toy like Dall E, think again. Sora is a data driven physics engineering. It's a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, intuitive physics, long horizon reasoning and semantic grounding all by some denoising and gradient mass. I won't be surprised if Sora is trained on lots of synthetic data using Unreal Engine 5. It has to Be. And he also talks about here the fluid dynamics of the coffee, even the foams that foam around the ships. And the fluid simulation like I just talked about, is an entire subfield of computer graphics which traditionally requires very complex algorithms and equations and of course the photorealism. So now, with what OpenAI have released here, I do think that this is genuinely incredible. This is by far something that I'm going to be continually watching again and again taking a look at it to understand exactly how on earth this thing works. Because I really want to see more videos from this. I really want to see what OpenAI have to say about this. I mean, they've stated quite a decent amount, but this is truly going to change everything if this goes mainstream. And of course with OpenAI, we know that they're working on a ton of different things. And of course with all of these clips, you can take a look at the prompts in the bottom of the screen if you wanted to know what they prompted it with. And I think this is about to shake up some entire industries because this is something that we didn't see. And one of the favorite things from this was of course this one right here, which essentially you can see that the prompt is actually a movie trailer for featuring the adventures of a 30 year old spaceman wearing a red wool knitted motorcycle helmet. And it's in a. I mean genuinely guys, we are going to get to a level in the future where we could probably generate entire movies with a single push of a button. And the crazy thing that people are missing about this clip right here is the entire consistency. Character consistency is something that we don't usually get because it's quite hard to do due to the randomness of the generative models. But it's clear that here, this is no problem for this model right here. And of course this one right here was the debut video that a lot of people did see on Twitter. So I'm glad I got to showcase a lot of these videos because I mean, it's just absolutely incredible what we're seeing here. And I, like I said before, I'm still completely mind blown. This is very captivating stuff, really captivating technology. And again, very, very surprising. And do you know why this is so surprising? It's because Dall E3 was really, really good. And of course there were other models that were better than it. So when OpenAI came out with this model, this was something that I don't think anybody expected and something that we're likely to see even advancements in the future. On. So, with that being said, if you did enjoy this, I'll see you on the next one.