This YouTube video (by LARRUZ) reviews OpenAI's SORA text-to-video model, shows demo examples (movie-trailer-like clips generated from single prompts, image animation, video extension, video editing and interpolation), and summarizes training/architecture notes from the accompanying research (diffusion transformer on spacetime patches trained on videos and images). The presenter highlights SORA's high fidelity (up to ~1 minute outputs), emerging simulation-like physical behaviors, current limitations/artifacts, and OpenAI's staged availability via red-team access and planned metadata for provenance.
Questions this video answers
SORA (OpenAI's text-to-video model).
Up to one minute (≈60 seconds) in the demos.
Red-teamers (for harm/risk assessment), per the presenter referencing OpenAI's blog post.
Chapters
Okay, so this was the state of the art for text to video models almost a year ago. And here is an example output from a new model from OpenAI which is generating a trailer of a movie using a single text prompt. Look at the details in the video, it seems like it's shot by a Hollywood camera. This new model from OpenAI is called Sora and they are creating videos from text. OpenAI is not the first company to create videos from text. For example, there is one Pika lab which is free to use. Then there is Gen 2 from Runway ML and both of them are amazing products. However, OpenAI took text to video generation to a whole new level. The details that you get in these videos is absolutely amazing and they are extremely coherent. And a plus point is that you can create videos up to one minute long. Most of the other offerings can create only a few seconds video. When you look at some of the examples on the OpenAI's website, it's extremely hard to distinguish whether they are AI generated videos or real videos. And just keep in mind, this video was generated by a single prompt. You can see the reflection of the subject in the water and people walking around. Now, with a single prompt you can create different angles, different movements within the video, which is pretty crazy. So if you look at this video, it seems like probably it's a part of a game, probably 3D generated. But this was generated using a single text prompt, which is mind blowing. Along with the blog post, they also released this research article, video Generation Models as World Simulators. This goes into a lot more details on how the model is trained and what different capabilities of the model are. According to this research blog post, they train text conditional diffusion models jointly on videos and images of variable duration, resolutions and aspect ratio. They are using the video transformer architecture that operates on spacetime patches of video and image latent codes. I'll highly recommend everybody to read through this article because it's, it has a lot of information, but let's look at a few highlights. So one thing that they have noticed is that the longer you train these diffusion transformer models, the better results that you get. So, for example, here's a base compute. So this is an initial version of the model that they train. It looks very similar to this video that we saw in the beginning. However, as you train the model longer and longer, you will get much better results. So this is four times the compute of the base model. And this is 16 times, which looks absolutely adorable. This model will be able to generate videos with different resolutions and different aspect Ratio, which is pretty great for training this model. They use a very similar approach that they have used for training the Dall? E3 models. So basically they created captions for different videos in the training data set and then they used those text description to train this model so that it can generate videos from the text description. And the results are very impressive to say the least. So for example, this is a woman wearing purple overalls and cowboy boots. And you can see both the purple color cowboy boots as well. And she seems to be walking on ice that represents Antarctica. Now this model can do a lot more than simple text to video generation. So for example, you can animate Dall E images using the Sora model. So here is the input image, the corresponding prompt, and then based on that prompt, this image is animated. Here. Here are a couple of other examples of animating images using text prompt. This one is pretty actually cool because the cloud shows up with SORA written in the cloud. SORA also has the capability to extend generated videos. So SORA is also capable of extending videos either forward or backward in time. And here are a couple of examples. So here you can see one video and that seems to be extended both in the forward as well as in backward direction. So now it seems like you will be able to actually generate much longer videos than those 60 seconds in the beginning that they have shown. So according to them, below are four videos that were all extended backward in time starting from a segment of generated video. And as a result, each of the four videos starts different from others. Yet all four videos leads to the same ending. Now here is another video. It will be actually interesting to see whether it can extend actual videos. So for example, if you have a small clip and you ask the model to extend that, if it has that capability, that is going to be pretty mind blowing. Now it can also do video to video editing, which is kind of crazy. So here's the input video. Car is driving, right? Here's a prompt changes setting to a lush jungle. Now you can actually play around with this. So this is settings in the 1920s in old school car, right? So the movement is exactly the same. However, the the overall scenario has changed, which is pretty great. You'll be able to recreate your videos in different styles that you want. Another capability is connecting videos. So you can also use SORA to gradually interpolate between two input videos, creating seamless transition between videos with different subjects and scenes composition. So here's an example of a drone footage and it's being combined with this butterfly. So look at the transition that it creates. So it converted the drone into the butterfly and now it's kind of transitioning into this video, which is pretty awesome. Like the. The transition was so seamless, it was extremely, like, hard to actually even notice that, which is pretty brilliant. And this is just based on the text input. Now here are a couple of other scenes. So here's the drone footage of an old church. This is also AI generated. And then this shows a Christmas theme and it transitions smoothly into that scene from the drone footage. This is absolutely brilliant. Okay, here's another one. So apart from the video generation, it also has image generation capabilities very similar to Dall E models. And I think this is probably not the most impressive capability that SORA has. They also noticed some emerging simulation capabilities since they use an enormous amount of training data. So the model seems to have learned certain behavior of physical objects as well as people. According to them, these capabilities enable SORA to simulate some aspects of people, animals and environment from physical world. These properties emerge without any explicit inductive bias. With 3D objects, etc. They're purely phenomena of scale. So the model has seen so many videos that it can. It's able to deduce different physical properties of different objects. Now, with models like Sora, it's going to be extremely hard to distinguish between what is real and what is AI generated. But when it comes to video content, keep in mind this is just the first iteration of the model. The next iteration is probably going to be more coherent, more consistent, and a lot longer videos. Now, it's still not perfect. You could get results like these. The chair starts walking by itself, right? So those kind of things can happen. But I think the next iteration will be able to take care of these artifacts. Now, the model is not available today. So according to the blog post today, SORA is becoming available to red Teamers to access critical areas for harm or risk. Now, this can potentially have really harmful effects. This, there's going to be a lot of misinformation, especially if you're not able to distinguish between AI generated videos. So this really could have some harmful effect. And that's why OpenAI is taking their time to red team the models and its outputs. And that's why they're sharing their research progress early to start working with and getting feedback from people outside of OpenAI and also give the public a sense of what AI capabilities are on the horizon. So this might become available in near future. They are also working on integrating some sort of metadata in the generated videos so that it's relatively easier to identify whether a video is generated by Sora or it's a real video, which I think is going to be a critical component of all the content that we see on the Internet. Well, I'll put a link to the videos in the video description. Do check them out. And after seeing this, don't trust anything that you see online. All right. So I hope you found this video useful. Thanks for watching and as always, see you in the next one.
