10 Things About SORA OpenAI Isn't Telling You

Sora the video generator by OpenAI released on February 15, 2024. And I've spent pretty much every hour of my life scouring the Internet and researching what else this could do. And there's actually a lot of things that weren't obvious. In the middle of all the hype that accompanied the release of this AI video generator, I studied the technical report on detail, watched all the YouTube videos, spent an unhealthy amount of time on Twitter looking for all the discussions and the little findings people had. Matter of fact, since release, I didn't even leave the apartment. If we haven't met yet, I'm Igor. I made it my full time calling to research what AI has to offer and how to put it to work in your everyday life. And before doing that with the AI Advantage, I had a video production company that operated for eight years in Central Europe. I helped clients with everything from corporate video trainings to directing smaller commercials and even shooting festivals, nightclub videos. When it comes to videography, I've really seen it all. And this stuff is exactly in the middle between technology and video production. So I can't wait to dive into all of this. All right, so without further ado, let's look at all the implications of SORA that you might have not been aware of right away. Okay, so first of all, I want to talk about the audio. Because SORA only generates video, right? All the examples we saw were muted without music or sound effects in the background. And a lot of people rightfully pointed out that, hey, in film it's really 50, 50 at the very least, it's 50% visuals and another 50% audio. And there's many layers to that, right? You might have the actor's voice voice as one track, but then there's also sound effects of things happening around them. And then you have foley, which is the background sound that just persists. You're not really consciously aware of it, but it's there. And if it's not there, the shot is missing something. So surely audio must be a complicated issue too, right? Well, not really, because 11 labs actually reacted to the SORA release and they released a new sound generator that from text prompts is able to generate an entire soundscape. Okay, so today we don't have access, right? But if OpenAI hooked up Sora to this audio generator, you would have an audio visual generator where you create full soundscapes, have a quick listen. And sure, a sound designer could do this manually, but again, if you're a one man show and you're producing a Commercial, like I did. So, so many times you're doing everything yourself, from planning to recording, editing, doing the sound design, doing the color grading, doing feedback rounds with the client, invoicing, and oftentimes you don't have budget for a sound designer, so. So you bet that there's going to be models, I don't know if Sora or others that combine both, they're going to give you audio visual output. This is not a question, that's just a straight fact at this point. And with tools like Suno AI out there already that can generate full songs, including lyrics, at a decent quality, with AI, well, you're going to be able to generate the background music, the background sound effects, the voices that are in the scene, because voice generators are a thing and they're virtually indistinguishable already, right? And now the video components. So we really have the full stack for audio visual production. It's just a question of time now. And from my estimate, it looks to be months, not years, till we get there. Okay, my next point is all about the capabilities of Sora that are actually brand new. Because a lot of the stuff that we saw just drastically reduced the cost of what it takes to produce a clip like this or an animated video like this. You might be aware that movies like this exist, right? It just costs a lot of money to produce this. So first of all, let's talk about the things that are actually brand new and not just a cost reduction, although that has its implications too, and we'll talk about that. But the things that are actually new are. First of all, you can extend videos, okay? So this is beautifully outlined in a technical paper here, and it shows the example of a San Francisco subway car. So as you can see, this clip is the same in all three instances, but if you back up a little bit, then extended the beginning of it. Okay, so as you can see, the video generated by Sora is different every single time, and it seamlessly transitions into the subway car. So. So this is something that was not possible up until now. Okay? It generates this video from scratch. Now, I guess you could argue that you could recreate this entire scene in 3D and create the frames before that and seamlessly transition into it. But you have to realize that at a certain point, this is going to become a feature in every editing software, right? You'll have just an image and it will turn it into a video, and then you can extend it to any duration. You can add a clip before, add a clip after. You'll be able to turn Your old family photos into video. Vivid memories, sort of. That is really scary. But it's going to be a thing. And you bet apps like Instagram at one point, I don't know when are going to have a feature where you're going to be able to turn a photo into video and then extend that indefinitely. Another new capability is you're going to be able to loop videos. Okay. And this is also something that you could kind of, but not really achieve today. Definitely not in this form. Okay. You'll give it a video clip and it will be generating extra frames that will seamlessly let the frame footage loop. I had a good chat with a friend and we kind of talked about how this could be the new Rickrolling on the Internet. Because if you do this to a longer clip, you just don't realize that it's looping and that it's just playing forever. So you could send somebody a clip and it might take them minutes to realize that the whole thing is looping and just repeating over and over again. Anyway, this is something that was not really possible and some people went ahead and tried this anyway. In videography there was this whole, whole trend a few years back where people were trying to seamlessly transition one thing into another. Like for example, and my shirt is gone. Magic. Now those are the simplest way to do it. But here we will have the capability of generating brand new frames and things will be able to loop indefinitely. Okay. So those are the new features you can expect in editing software somewhere down the line. But then there's a lot of the ones that are a simple cost reduction. This is why people refer to it as the death of Hollywood in many cases. Now I don't know if that's an accurate assessment in my opinion. I think they're going to use this tech to their advantage to lower the prices of production and pump out even more content. We'll also talk about that soon. But let's finish up the segment and talk about the things that were already available. But now it's just a 10,000x reduction in cost for that calculation. I seen a subscription price that is somewhere around the GPT plus plan. So what's going to be possible at this super low cost is first of all generating images. We're able to do that with other image generators, right? Sure. These are hyper realistic and very high quality, just like midjourney and so. But. But then it's the capability to turn images into videos that is very, very big in my opinion because it's going to make it so Easy to craft compelling videos. Like, I feel like most people that talk about this don't appreciate how much this is going to lower the barrier for entry for videography and high quality videography. That is because you're going to get access to things like this. So even if you've seen this before, I think I have a bit of a different perspective here. So look, here on the left you have the drone image. Here on the right you have this butterfly, right? And here in the middle you have the mix of the two where the drone is flying through something like the Colosseum and then it morphs into a butterfly. And look, I could do this day. Okay, this just takes about three to five hours of work, depending on your skill level. You just go into After Effects and you rotoscope out this butterfly, meaning you go frame by frame. That's 25 frames every single second. And you make sure you animate a mask exactly in the form of the butterfly's wings. And you redo that for every movement. Now, yes, there's tools that help you, but a lot of times you're stuck with manual labor there. So it might just turn out that the three to five hour task turns into 15, 20 hours person. And then you can bring the butterfly into here and morph it into the drone with something like a morph cut inside of Premiere Pro. Now if none of that means anything to you, that's fine. I'm just saying hours of work are going to be done like this. And this is just one simple example. In many others, a one man crew could never do this, right. All these animation related examples where they turn an image into an animation like this are usually just unfeasible for a one man show. It takes too much time to animate all the little things. You might be able to do it for a few shots, but if you do a whole one minute trailer, you'll find that you spent two weeks at the computer. If you really animate all the little details like in this shot. And you have a lot of different shots. So that's my second point. It lowered the bar by a factor that is larger than most people realize. I don't know if it's a thousand x or ten thousand x, but a lot of these things were unthinkable for small crews or one man shows. And now they will be doable. Like for example, before, after. Okay, so this point is all about the editability of the video. And here on Twitter, Owen Fern went ahead and he criticized the fact that, hey, yes, these generations are absolutely Incredible. But what if the client has feedback? And this is very, very appropriate criticism in my opinion, because clients always have feedback. And if you're going to use this for job, if this is supposed to be the death of Hollywood, just between directors and producers, there is so much feedback going on in the post production of any advertisement movie. Heck, even if it's an event video, I had clients that went back and forth 10 times and gave feedback over and over again and I had to adjust things. So Owen points out here that, yeah, there's going to be a lot of little details that will need to be changed about these scenes. And with Sora, you're not really able to go back and change little details. Right? You're going to have to regenerate the whole scene. And maybe you like the character here, but you just don't like the fact that this, this is not a thumb. It just looks like a fifth finger. And we would like to give it a look of a thumb. Can we do that? And his point is the answer has to be no. And then you have a dissatisfied client, which is a very fair point. But as I've been following this very closely over the last months, there's one tool and one research that needs to be pointed out here. Okay, first things first. Runway ML, the previous, so to say, leader in AI video, a few weeks ago introduced a feature called Multi Motion brush tool which allowed you to use multiple brushes on the video to just animate specific parts. Now that is for animation, but over in midjourney and many other image generators, you're able to do something called inpainting, where you just paint in a little part of the image and then edit just that you can reprompt it. So on images today, you could actually go in and just paint in this little thumb and say, regenerate the thumb. Why would that not be possible on video? Eventually it will be. And further than that, ByteDance, the creator of TikTok, actually published a research paper like less than a week ago about this so called boximator. Okay, so I didn't cover it on the channel because I like to cover things that are available today or truly, truly revolutionary. This kind of falls in this in between zone of, hey, really interesting, but it's not available and in my eyes, probably not worth a dedicated video. But look, the whole point of this is you draw different boxes in the scene and thereby you can control the scene in great detail. So if you select the balloon and say it's going to fly away in this direction and then you select the girl and she's going to run in a different direction. Exactly, exactly. That is going to happen. So between tools like the boximator and inpainting in midjourney, it's just a question of time where you're going to be able to use a mix of these tools and also inpaint on top of AI video. Now, sure, there's going to be a temporal axis there, right? Because on images you only have the x and Y axis and in video there's also the time axis and sometimes you even have movement in Z space. But between this research and inpainting, I can totally see that happening for AI video two down the line. Plus, as we know with prompt engineering today for language based models, there's a lot of control that you have in the text prompt. You just have to be really detailed. If you look at a lot of these prompts, they're good, but they're not as detailed as they could be. Some of the best stable diffusion prompting is extremely detailed also in midjourney and stable diffusion, if you keep your prompts relatively simple, you're going to get varied results. Even if you reroll the dice and create a new scene, it's going to be very similar. Plus, let's refer back to midjourney again. Digest recently announced a new character tool where it's going to maintain character consistency based on a character that you pick in a tool. So all of these AI image features that we've been talking about and I've been tracking regularly, they're going to apply to video tool. It's just going to take longer. But I absolutely believe that we'll be able to implement all of this little feedback into AI video and therefore this actually being production ready at some point. Okay, so my next point here is that I didn't expect right in the beginning is that you can prompt stories into existence from a single prompt. Okay, so here's an example from Bill peebles from the OpenAI team and he generated an entire story of two dogs that should walk through NYC. Then a taxi should stop to let the dogs pass a crosswalk. Then they should walk past a pretzel and hot dog stand and finally they should end up at Broadway signs. And if you follow this channel, you might know how much context you can add to text prompts to achieve exceptionally accurate results from things like ChatGPT. If you added way more details here, I believe they would be reflected in it and then the story can develop. And as right now you already have tools that can manipulate someone's mouth to speak in another language so it looks naturally also. That will be possible here. So you will be able to create these long shots like they have in movies, which are incredibly difficult to achieve. I mean, some movies like Dunkirk took it so far where they turned the movie into a single take. It all flows seamlessly and Sora is able to do it too. And that I didn't expect in the beginning. Also, they didn't share this example right off the bat. I think this is actually very, very impressive. And if now we're already able to generate stories from, from a single simple text prompt, it's just a question of time until we arrive at something like this where you just type in a prompt and you get a full movie back or a full show. I mean, at some point it's just a question of having enough GPUs. This is obviously just a mock up, but something to think about, especially because this is the worst attack is ever going to be. And you know what, let's talk about that point. That is actually my next one. So where are we in the timeline of this? Okay, it was really helpful to look into some of the discussions that are happening online to orient myself in terms of where we actually are today. So Emad Mostak from Stability AI actually had a fantastic take here. He compared this to the GPT3 of video models. So if you didn't know GPT3 was the predecessor to ChatGPT, okay, it was available before, but the interface was not as intuitive and you actually had to prompt it differently rather than ChatGPT that had the reinforcement learning for human feedback, which means a lot of humans feedbacked the outputs to make it more user friendly for humans. And that's where this is at right now. Okay. It's not at the ChatGPT point where it's going to be really easy to use and it's going to gain mass popularity. And then we got GPT4 and all the additional features and it's just crazy capable now. And he even said that all the image generators like Stable Diffusion, were more comparable to GPT2, where the quality of the outputs was not nearly as good as GPT3. So as in large language models, this puts us on the timeline somewhere in the middle of 2022, because the ChatGPT, GPT4 llama and Mistrals will come over the next few years. We're mom's at the pace that we're moving ahead. Right? And on this topic, there's another fantastic thread by Nick Sampierre here on X and he ran all the exact prompts that Sora generated through my journey and then paired them with the results. And the thing is, they're shockingly similar. Right? So people are already joking that, hey, is my journey just OpenAI disguised? Probably they're just using very similar training data. Right? But look at that. All of these examples are very similar. Now I'm sure these are the ones that were the most similar, right? To create this illusion of it essentially being the same model here. I mean, if you look closer, the beaver is very different. But the point is these are not night and day, Right? Sure, these helmets are completely different, but the cinematic look is very similar with slightly different color grading down here. Fair. But the point that I'm trying to make here is that we literally skipped two to three years ahead in AI video because what we had up until now was something like GPT1 or GPT2. Oh, that's hot. Now we got GPT3 that is actually usable and can create useful outputs that are essentially hyper realistic. But we're not Even at the ChatGPT moment yet where you get editability and things like audio generation that we talked about here. That is all yet to come. But again, at the space of development, we should probably be thinking in days and weeks and maybe months and not years or decades. I guess that poses the question, at which point in the development do we reach the Matrix? And I don't know the answer to that question. I'm turning 30 next month and it does feel like it will happen in this lifetime or something akin to that, right? Who knows? Moving on. Okay, so my next point goes back to my original video where I stated that, you know, this is going to be the death of stock footage. I sell it myself since almost a decade and there's just no way people are going to be paying 50 or $100 per clip if they can just generate them for a few cents. And yeah, I think that one is an obvious one. But beyond that, it really got me thinking about what this means for video video creation, especially for the smaller crews and one man shows. Well, you're going to be able to generate entire video libraries for yourself. Hear me out. So right now, if you have a video, let's say this is the A roll, right? This is the main story of the video. Me talking, presenting to you all my findings. And then on top of that, we have something that we refer to as B roll. These are the clips that are there to add an additional layer of information. They add visual interest, keep, keep you more Engaged and really allow us to get the most out of this audio visual medium. And right at this very moment, you're consuming both audio and video at the same time. So we're trying to make the most out of all these layers. I do my best to keep my speech and presentation concise because I value your time. And then in the editing, we do our best to add as much information on top. And right now that is done through B roll. So we pay for various libraries where we take these shots that enhance our videos. And we also pay for various music libraries to add the right type of music to enhance the atmosphere of the video. But with models like Sora, this will really change the game because you're going to be able to generate an entire library for yourself for that specific project. Because the cost goes down so much, you're going to be able to prompt things into existence that beforehand you would have to research, download and compile. And usually they don't even match. And you have to do color correction and color grading on top of them. And here, as you can see from a single text prompt, we got five video frames. And all of these can be upscaled with something like Topaz Video AI, right? That tool is paid, they cost a few hundred dollars, but you can upscale 1080p clips to 4K with AI really effectively. But here you're just going to be able to prompt them. And then again, just looking over at all the AI imaging tools, all the features that we see in the imaging tools are going to be available to the video tools. So something like a one click upscale to 4k quality is going to be there. Can you regenerate this or can you generate four more just like this is going to be there? You can think about the whole midjourney interface in Discord being something that you can do with these videos, upscale, reroll more like this, use a different version of the model, and after a few minutes you'll have a whole library of B roll that can enhance your video. Now, I, as a video creator, can't wait for this. I know that eventually the end point of all of this is the technology really replacing a lot of content. And who knows if I'll be sitting here and presenting the news to you, if an AI can do it in real time, minutes after the release of something, and you will be able to get it exactly, exactly in the voice that you prefer, while it also respects your context, right? So in this video I kind of have to assume your knowledge level, right? So at certain points I also have to Assume that somebody never created a video before. But some of you might be experienced directors that know all these concepts and know how the industry works. Well, the AI is eventually going to be able to create that exactly for your context. But I digress. The point here is that at least for the footage, at least for the production of this video, I could have a custom library that is going to enhance all the visuals. And maybe we could be taking a trip through Tokyo as of now, where I present these ideas. There's going to be some point where I'm just going to be able to take my voice and use my digital avatar, let him walk through Tokyo and explain these concepts in a very practical manner without ever leaving my desk. I don't think at this point that is a stretch. A week or two ago it seemed a bit unreal to think of lifelike video. The best we had was animations that were good and talking head videos that looked okay. They looked convincing for a second or two if you weren't looking for AI. But again, if this is the GPT3 of AI video, then what is the ChatGPT and the GPT4 going to look like? That's what I'm already thinking about. And some of these advanced capabilities are outlined in the technical paper too. Here it clearly states that you're going to be able to create videos in any format. Okay? So from 19201080 to 1089 20, so you know, phone formats all the way to widescreen and then cross cropping into cinematic formats from this is easy, right? All you need to do is add black bars at the top and bottom and you have all the cinematic formats. So really there's going to be a lot of variability and you're going to be able to get exactly the B roll that you need for your project. And then eventually AI is going to be creating the scripts and editing the video itself according to all the other videos it saw and how they were edited. Right? I mean, that might take a lot of time and we do so much manual work with these videos that there's always going to be a style expression and a handwriting to the post production of a video, I think. But it's crazy to see that. You know, a week ago, thinking about the fact that you would have a library of B roll for a specific video, well, you had to go out there and shoot it in the real world or you had to purchase stock footage and then it was scattered and all over the place. Here you're going to be able to get the best of both worlds. Going to get great B roll and all from the same scene and it's going to cost virtually nothing. Or if you have some B roll that you already use, going to be able to extend that, or maybe you have some phone pictures and you're going to turn those into, into B roll. It's really a whole new world for video production. I can't overstate that. But it doesn't end there. And this brings me to my last point, which is 3D worlds and world Generation. Because in the technical paper they actually refer to this as a world simulator. And I think that's a big claim, but it's also a justified one because if you take some of the clips at face value, it's incredible. It's temporarily consistent. These houses are not warping, right? You're moving through the scene like a drone would. You have these people on their horses going about their daily business. It's incredible. But what you have to realize is that beyond that, you can apply this in something like Gaussian splatting, which simply put, is a technology that creates a so called Gaussian splat that is the 3D representation of the video. In even simpler terms, it turns a video into a 3D model and this is what it looks like in practice. Now look, this is a simple video that it wasn't even intended for, for this purpose. But you could easily imagine a drone shot where the drone parallaxes around the subject and gets it from all angles. And then you can create 3D objects of something that doesn't even exist. So right here, Manovision took exactly this drone clip and he recreated it as a Gaussian splat and then brought it into Unity, a real time game engine. And then you can animate the camera and insert characters and do all sorts of things. Right. The important fact here is that Sora doesn't have to do everything from A to Z. You can still have a human write the script. You can still have a human in front of a green screen acting it out. You can have your favorite actors in these scenes. But it's going to be so much cheaper to produce because you're just going to generate all the environments like this. And then everything is going to be shot in front of a green screen until AI perfectly synthesizes the actors voices, which if you follow this channel, you know that it already has. And then the last missing piece is really the human part. It's character consistency and the ability to edit little details so it aligns with the vision of everybody involved in the movie's. Creation. And then if you take that thought experiment even a step further, you end up in Minecraft. Because in the technical paper, you can see these that are not recorded from within Minecraft. These have been generated by Sora by simply including the word Minecraft in the prompt. It saw so much Minecraft footage that it was able to recreate Minecraft perfectly. And if it can do it with Minecraft now, how long until it will do it to all of this world? I don't know. But I'm scared and excited at the same time. But one thing is for sure, I want to stay on top of all of this. I'm going to keep my eye on it. And if you want to follow me along for the ride, subscribe to this channel subscribe to our weekly newsletter that is completely free and keeps you up to date once a week with all the revolutionary breakthroughs. And that's really all I got for today. Except if you want to try out Sora, there is actually a very, very limited demo here on this page. If you haven't tried this yet, I recommend it because it's the closest you can get to trying it. And it's this little interface here where you can change these variables so you can go from an old man to an adorable kangaroo. And then there's a few more variables that you can change out here. Okay, Antarctica. And for now, this is the closest we get to playing with this thing. So I hope you enjoyed this. Let me know which one of these was new or interesting to you, and if you have even more facts that I might have not considered yet, also leave those below. And if you haven't seen the original video about the announcements and all the video clips they presented, that is over here. All right. I can't wait to see how this develops and what the competition comes up with. This is a whole new world, and I'm here for it. See you soon.