A YouTube video (Two Minute Papers-style) by Dr. Károly Zsolnai-Fehér showcasing OpenAI's new text-to-video system, Sora. The speaker evaluates Sora using three criteria—visual quality, temporal coherence, and fidelity to prompts—demonstrates creative examples (e.g., themed corgis, virtual worlds), and sketches a technical explanation (latent-space synthesis and heavy compute requirements). He emphasizes the system's high quality (full HD output), consistent object permanence, potential to transform existing videos and synthesize playable virtual worlds, and calls attention to rapid future progress.
Chapters
Buckle up, fellow scholars, because what you are going to see today is something that might be the craziest thing I've been able to show you in more than 800 videos. This is the kind of video an AI could create yesterday. And today it can do this. Holy mother of papers. Yes. OpenAI just released their own text to video AI so sora. And it is so far beyond anything else I have ever seen. It is hard to put into words. Dear fellow scholars, this is two minute papers with Dr. When I first saw these results, I thought this was some April Fool's joke. No, this is not a video coming from a real camera. This is a video that was synthesized pixel by pixel by a new AI. So. So let's give this one a spin. These AI videos we will evaluate by three criteria. 1. Quality. This is shocking. The quality of these works is out of this world. If we are not actively seeking for errors in the footage, in many cases, we may not even know that they are made by an AI. And it gets better. Their Dall E3 system that is an expert at creating images. And I just stop these videos here and there. And the still images are often as good or even better than what Dall E3 can make. This is beating the king in its own game. Unbelievable. 2. Temporal coherence. This means that the AI understands exactly how each image in the video should follow each other. This is what it looks like if you don't have temporal coherence. A paper from just a few years ago. And now we have this once again, temporal coherence. Second to none. Wow. And three. Wait. This may still not be a great technique. Now I hear you asking why is that? Well, it has to follow our prompts correctly. It has to be true to what we asked for. You see, there are techniques out there that give you really good quality coherent videos. However, they don't care too much about the prompts that we write. And what about this technique? Goodness. That is exactly what the prompt is asking for. I am out of words. But it gets better. It even has a hint of imagination. For instance, we can ask for a Corgi that's also a vlogger, an utter on a surfboard, an Italian pup, you name it. Just ask and it will do it. Imagination in a machine. What a time to be alive. Hmm. Wait, I just noticed that we need to take a look at a fourth thing from now on. And that is object permanence and consistency with previous techniques. When something got occluded and is now visible again, the AI might not Remember? And it might look completely different, but here, let's see. Wow. This has a consistent world model. So much so that even when we move around in 3D space, everything remains where it should be. And this can do so much more. We can even transform an existing video into a completely new one by just writing one text prompt. And now hold on to your papers, fellow scholars, because it can also synthesize virtual worlds. Whether that will be something that already exists, like Minecraft, or a completely new game made from scratch, up to you. Just one more paper down the line, and it might be that you don't even need to develop your own games. You can maybe just hook up a controller, write a text prompt, and OpenAI Sora will give that game to you immediately. So how does all this magic work? Well, one of the key ideas is that the synthesis takes place in a latent space. What is that? It looks something like this. This is one of my papers where you can walk around in this 2D latent space. And each point in this space represents a material for a virtual world. And here comes the key. The latent space works well if you can guarantee that exploring a nearby point, you get similar material models. The link to the paper is available in the video description. And this concept also works for creating new fonts. And now to create new videos too. And they come in full HD resolution. Wow. So is this concept any good so far? Well, let's have a look. Wait a second. That is not even close to what we've seen. What happened? Well, one word, compute happened. You see, if you don't have enough computational power, this is what you get. If you have four times more, you get this. And if you have 16 times more, you get this. Oh, yes. So the concept comes alive only with a sufficient amount of compute. The virtual brain, if you will, has to be developed enough to imagine all of these videos in high quality. And my goodness, this is perhaps the biggest jump in quality between two research works that I have ever seen. And this video series has been around for more than 800 episodes now. And now it's time. Time for what? You ask? Of course it is time to invoke the first law of papers. The first Law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. And here is the one more paper down the line. Now exercise. Leave a comment about what you think we will be able to do. Just two more papers down the line. I'd love to know what you fellow scholars think, especially now because once again we share one of those sweet moments where we witness history in the making. In his excellent video, which I highly recommend, MKBHD says that since this is trained on videos made by humans, it likely cannot go beyond what it had seen from humans. I would like to note that in some cases we see AI papers that have proper zero shot performance. What is that? This means that leaning on all this knowledge like a human, it can try to create new things it hasn't seen before. For instance, you could ask for a new kind of vehicle for T. Rexes and it could infer that T. Rexes have these tiny little hands, so it would have to have a wheel that is suitable for their little hands. We will be able to test that and so much more and more as soon as it is out there. And we will soon be back with a video on a different AI video system that is not as good as this, but it is more controllable and it is something that you will be able to try out for free right away. We will also have a more in depth video about the capabilities of this new technique soon too, so make sure to subscribe and hit the bell icon to not miss out. Experiment tracking, model evaluation and production monitoring for your deep learning projects and LLM apps. This is what weights and biases does and it is the best. Everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
