A short-form YouTube explainer by Dr. Károly Zsolnai-Fehér (Two Minute Papers) summarizing a new NVIDIA text-to-video model. The presenter showcases many examples (artistic styles, natural phenomena, character-driven clips, driving simulations), highlights strengths (camera motion, ~2000×1000 resolution frames, up to five-minute clips, single-image future prediction), explains the core method (diffusion-based generation with a temporal fine-tuning step to enforce coherence), and notes current limits (occasional failures like a koala playing piano). The video closes with a sponsorship plug for Weights & Biases and an invitation to subscribe.
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Finally, as everyone knows, that text to image AIs are capable of creating incredible photos, digital art, whatever you wish. Now we seem to be conquering video too. What? So we just write something and exactly that video comes out? Doesn't that sound impossible? That is so much harder. You see, images is one thing, but videos require a much greater understanding of the world around us, so much more computation and one more secret ingredient that I'm going to tell you about in a moment. So after Google has published Image and Video, now Nvidia is here with their text to video AI tool. And it's not just real good, it can do things that other systems can't do yet. So first, examples of course we start out immediately with a scholarly example of two pandas reading a paper approved. It is also excellent at time lapse videos. I love this one. We can also let our imagination run wild with this one. For instance, we can even ask for a stormtrooper vacuum cleaning a beach. Very productive. Artistic scenes work well too. Here is the look out of a rainy car window, but in the style of a Van Gogh painting. Apart from a little flicker, this is excellent. Now what about natural phenomena? You know that my favorite is fluid simulations. So we have to look at one of those two. Hmm. This is easily good enough to make me crave coffee. The swimming turtle is also a delight to look at. And these are not just some tiny stamp sized videos. We are getting a sequence of approximately 2000 by 1000 resolution images. Wow, that is incredible. I also loved how well this new paper can deal with camera movements. Flying into a fantasy landscape Worked really well. And I feel that this rotating camera around the grapes is just one paper away from near perfection. Wow. And these results are already very impressive. But hold on to your papers fellow scholars, because you have seen nothing yet. This can do so much more than just text to video. With this we can use our own characters to create a movie out of thin air. You see, here is a bunch of images of the test subject. Kermit, is that you? Alright, now let's ask the AI to make it play the guitar. That is fantastic. But that's not the way of the true scholar. Of course, the way of the true scholar is writing research papers. So can it do that too? Oh yes, yes it can. Good job little AI. It can also generate these driving sequences. So what is that good for? Well of course to feed hypothetical situations to a self driving AI so it can have a look at footage and practice Safely within a simulation before bringing its knowledge into the real world. Loving it. But wait, these are all quite short sequences. So what about video length? Well, worry not a second about that one. It can generate videos up to five minutes in length. By the end of that example I saw at most a tiny bit of degradation of quality or even less. And get this, it can also predict an entire video sequence from just a single image. That is also excellent for self driving cars. Why? Well, just imagine giving it a starting scenario and it can simulate hundreds and hundreds of potential variants of this situation to train these AIs. What a time to be alive. So how does all this black magic work? Well, let's pop the hood and look inside. Oh yes, this is going to be excellent. I mean, not this, the one after this. You see, this is a diffusion based technique with which means that it starts out from a piece of noise and gradually reorders these pixels to form an image. But there is a problem. Do you see the problem? These images are completely fine, but they don't form a coherent video. However, look at that. After the new proposed temporal video fine tuning step. Look, now we're talking. This is truly a coherent video in the making. Loving it. And that temporal coherence is the secret ingredient that I promised that makes all this work so well, so good. Now not even this technique is perfect. Asking a koala to play the piano is a bit too much. However, I bet that the first law of papers applies here. The first law of papers says that that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. Also, by the way, I just came across this. This shows the length of 2 minute papers videos over time from the first episode to 700 and beyond. It used to be shorter. Can't argue with that at this point. It's a running joke that two minute papers is never two minutes. But I wanted to ask you, what do you think? Should these be shorter? Let me know in the comments below. Now, I haven't found the source code for this paper yet. However, there will soon be an episode about a way for you to try to create videos like this soon. If you're interested, make sure to subscribe and hit the bell icon not to miss it.
Weights and Biases provides tools to track your experiments in your deep learning projects. What you see here is their amazing sweeps feature which helps you find and reproduce your best runs. And even better, what made this particular run the best. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
