NVIDIA's New AI: Better AI Videos Are Here!

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. In 2021, scientists at Google had an absolutely insane idea. Do you see this photo? Yes. They said now let's give this to the AI and ask it to fly into this photo. Insanity. And yet it worked like this. Absolutely amazing. It was not given a video and not even a collection of photos, just this one photo. However, there is a problem. Do you see the problem? Well, this is the problem. Yes. If we look for a while it has to come up with newer and newer content and over time the consistency of the results suffer. We don't get one coherent landscape but something that morphs into something else that not connected to it. And today this is why I am so excited to show you Nvidia's new paper on video generation. Let's see. Yes, they promise exactly what we are looking for. Better long term consistency. So first let's see what previous techniques can do and then what this new one can. This is an excellent test case. Now you may be asking what in the world is this? And I say that is exactly the point. Mokogan HD is a technique from 2021 from just two years ago and we cannot even tell what it is trying to do. Here is a later work from 2022, the time agnostic Time Sensitive Transformer Tats. In short, this isn't great either. But now at least we see what it is trying to do and that is rendering clouds. It starts out reasonably well but as you see the results quickly start degrading. And that is the point of today's video. Long term consistency is super difficult. Dia again, another work from last year does way better than these previous two. Great progress. But as you see the video is still full of artifacts. And now hold on to your papers fellow scholars and let's see the new technique. Holy mother of papers. Do you see that it is leaps and bounds beyond any of these techniques? And all this just one more paper down the line. How cool is that? I don't know about you but I am itching to see more examples. Lets compare against Stylegan V. One of the state of the art techniques out there for this task. First, horseback riding. I must say it is pretty incredible that today an AI is capable of creating a video like this. However, there are issues. Things morph in and out of existence. And did you notice this doesn't really synthesize new scenery. It just repeats this jump over and over. Now let's see the new technique. Wow, now we're talking. After that first jump we get new scenery as we enter the forest. But it has even more advantages. Look, when synthesizing a landscape, we have a much better sense of camera motion and rotation with the new technique than with the previous one. And it can do something even better too. Yes, this is going to be everyone's favorite super resolution. What is that? The enhanced thing. First, it generates a coarse video and then adds more and more detail to it until it becomes a much sharper, higher resolution video. Just look at the difference. It takes no more than a heap of pixels and that looks like a computer game from 25 years ago. And it understands what this scene is meant to portray and it creates this detailed version of it. And I would argue it would do at least as well as a human would. Wow. So clearly the new technique seems way better. But this is a research paper in which scientists at Nvidia also have to show mathematically that it is better. So how do they do it? How do they measure the differences? Well, first, at the risk of simplifying the measurement, we show these videos to a neural network that somewhat mimics the perception of humans. And the lower the score it gives, the better. And oh my goodness, look at that. Apart from a few super low resolution cases, this new technique is so much better than the previous methods. Incredible. But there is another way of measuring which is exactly better and even more importantly, by how much. And that is of course showing these videos to humans and asking them. In the user study, 500 people were asked which is more realistic. And about 80 to 85% of the people asked chose the new technique over Stalgan v, the previous one. That is incredible. And once again, all this progress in AI research in less than a year now. Clearly the new technique is not perfect either. It has artifacts, morphing and other issues as well. However, this is incredible progress in just one paper. And as always, as, as a wise fellow scholar, please invoke the first law of papers which says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. So finally, better long term consistency for AI generated videos. What a time to be alive. Also good news, the source code, datasets and pre trained models for this paper are available. So let the experiments begin. So what video would you generate using this technique? Let me know in the comments below.

This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights and weights. And biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning, and even hyperparameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link wnb me paperintro or just click the link in the video description and and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back.

Thanks for watching and for your generous support and I'll see you next time.