OpenAI Sora: A Closer Look!

This is a closer look at Sora OpenAI's amazing text to video AI. We already know that it can create amazing videos from your text prompts with unprecedented quality. It is a huge leap in capabilities, but it can do so much more. We know that it can take a still image and extend it forward into a video. But get this, it can also do the same, but backward. And this one comes with a twist. We prescribe how the video should end and it writes several possible ways of getting there. And they all feel completely natural to me. That is awesome. I love it. Dear fellow scholars, this is Two Minute Papers with Dr. Caro Zsolnai. And they really know how to make me happy because I am a light transport simulation researcher. Buy trade, that is ray tracing, if you will. So looking at the glossy reflections on the road and how they change over time makes me really, really happy. Just look at that. And not just the quality, but also the long term coherence as well. I am used to looking at papers that can create videos that are at most five seconds long. And this particular one is a 20 second snippet. That is amazing. But I read that it can go up to 60 seconds as well. And look at those beautiful refractions. And even little specks of dust and grease marks on the glass are synthesized. Wow. Now clearly there are lots of mistakes here. We can all see that. But this is also a remarkable sign of understanding the world. It is. And it can generate a full scene in one go. We don't need to splice many results together. It can really give us a full scene without any cuts. Absolutely incredible. But it gets better. This result puts even the 62nd to shame. Let's look at this together and if we do it for a while. Wait, so when does this end exactly? Well, did you notice? Yes, you are seeing correctly. It never ends. It can also create infinitely looping videos as well. And it can also perform limited physics simulations. We are going to get back to that in a moment. That is not only beautiful, I mean, just look at that. But this has a profoundly important story behind it. Now, it can also create still images. Yes, I hear you asking. Why is that interesting? We just saw videos. Those are way more difficult. Well, but these are 2048 by 2048 images. Those are huge, with tons of detail. And remember, Dall E3 appeared approximately six months ago. That technique is a specialist in creating high quality images. And this one even beats Dall E3, the king in its own game. And it does all this as an almost completely unintentional side effect. Wow. And large language models think in tokens, batches of letters. But SORA is for video, and it does not think in tokens, at least not in the way the language models do. It extends the definition of tokens to visual content. They call them patches. Then, when the neural network sees a RAW video, it compresses it down into a latent space where it is easier to generate similar new ones. Now, wait, what is that? Well, a latent space looks something like this. This is my paper where you can walk around in this 2D latent space. And each point in this space represents a material for a virtual world. The key here is that nearby points generate similar materials, making it excellent for exploring and generating new ones. The link to the paper is available in the video description. And now imagine the same concept, but for video. Now our scholarly question is as so what did this neural network learn exactly? To be able to understand that, we have to look back to GPT2. GPT2 was an early technique that read lots and lots of product reviews on Amazon. And then it was given a new review, split in half. And then it was little AI continue it. Then the AI recognizes that this is not gibberish text. It has structure, it is in English. So to be able to continue this, it has to learn English. But it got better. It also needed to know whether this is a positive or a negative review. Did the user like the product? If it seems like it, from the first half, you need to know that to continue it. So it learned sentiment detection as well. And the incredible thing is that no one told what it has to do. It learned this all by itself. So now if we give it an image, it has to extend it forward as a video. That is simulation. To be able to simulate the world, it has to understand the world. And for that it has to learn the underlying rules. The world contains plenty of liquids and smoke, so it had to learn fluid dynamics as well. And once again, it learns physics. As a completely unintentional side effect. It learns it because it has to. The architecture they use for this is called a diffusion based transformer model. When you do text to image, the model starts out from a bunch of noise and over time reorganizes this noise to make it resemble your text prompt. However, here we are talking about video, not just one image. So easy, just do it many times after each other, right? No, not at all. You see, if you do that, you get something like this, you get this flickering effect because the neural network does not remember well enough what images it made previously and the differences show up as flickering. You can't do that. What you need to do is create not just one bunch of noise, but many, many bunches and refine them at the same time while taking into consideration not just their neighbors but every single image. And this is how you get long term temporal coherence and OpenAI Sora just nailed it. Bravo. And don't forget, as we add more and more compute, it gets better and better. It only comes alive in the presence of huge computational power. So I hope this gives a closer look at what it can do and and how. Not so long ago this was the best video we could do and now we have this. Can you believe this? This is the power of human ingenuity and the power of research. And just imagine what we will be capable of just two more papers down the line. My goodness. Subscribe and hit the bell icon if you don't want to miss out on it when it comes. You can count on me being here and and flipping out. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.