A YouTube explainer (by LARRUZ, uploaded Apr 21, 2023) on Nvidia’s new text-to-video research that converts pretrained image diffusion models (Stable Diffusion variants) into temporally-consistent video generators via temporal fine-tuning. The model produces 24 fps clips of 113 frames (~4.7 seconds) at much higher resolution than prior work (up to 1280×2048) and demonstrates plausible physics (e.g., milk diffusing in coffee, burning wood). The speaker notes strengths (smooth temporal coherence, high-res outputs, DreamBooth-personalization for turning images of a character into full videos) and weaknesses (fails when camera and subject move together, artifacts on legs/motion). Practical uses highlighted include synthetic driving data, driver-behavior simulation, and potential for realistic deepfakes; authors warn about ethical/safety and dataset concerns.
Questions this video answers
April 21, 2023.
24 frames per second.
113 frames; ~4.7 seconds long.
Up to 1280 by 2048 pixels.
Total model size 4.1 billion parameters; 2.7 billion parameters trained on videos.
Chapters
Let's say you have a character and you just have access to the images and pictures. Using this model, you can make them do anything you want. In the distant past of few weeks ago, this was the state of the art for text to video generation. But now Nvidia, the chip maker that drives innovation in AI, just released this new paper that takes text to video to a whole new level. In this video we will look at some of the cool demos from this paper as well as some unsettling warnings that the authors have. Is a new technique to achieve temporal consistency. That means that the subsequent frames in the video have interdependencies and as a result the output video is pretty smooth. With this approach now they are able to generate videos with a much higher resolution up to 1280 by 2048, which was previously not possible. In this work they are using stable diffusion based models to create high resolution images using their latent space and the results are pretty amazing. In this video we will look at a few examples as well as some practical applications where this approach can be used. Here is an example of a teddy bear playing an electric guitar. High definition 4K. If you look at it, it's pretty nice. The colors are pretty good. The image definitely seems to be generated by stable diffusion based models. In some cases. If you're not paying very close attention, it's really hard to even figure out if these videos are artificially generated. For example, look at this time lapse of clouds at sunset time. It looks really realistic. Here is another one. A close up of grapes on rotating table. High definition four camera. If you look at it, it looks really realistic. This the texture is consistent for each grape. The color remain consistent as well even though the table is rotating. Now keep in mind that this doesn't work in all cases. There are definitely cases in which the model fails. We will look at a few examples later in the video. But how does it work? In this work they converted a pre trained image diffusion models into temporary consistent video generators. So initially different samples of a batch synthesized by the model are independent. After temporal video fine tuning, the samples are temporarily aligned and form coherent videos. Now these videos are generated at 24 frames per second and each video has 113 frames which results in 4.7 seconds long video clips. The model has 4.1 billion parameters, but only 2.7 billion parameters are trained on videos. It also seems to understand laws of physics. For example, in this case the prompt was milk dripping in dripping into a cup of coffee. High definition 4K. In this case, the way the Milk dissolves in the coffee is what you would expect in real life. Here's another example. In this case, the prompt was wood on fire. In this case, how the wood deteriorates with fire is what you would expect in real life. This is pretty neat. There are limitations to this approach and there are cases in which this approach will fail. I have noticed that when there is movement of the camera and movement of the subject at the same time, this seems to have difficulty in generating videos. For example, an astronaut riding on a horse, high definition, 4K. In this case, you can see that it mixes up the legs of the horse and the movement is not really great because both the camera as well as the subject is moving. Here is another one. A stormtrooper riding a bike in Sunset Street. 4K high definition. Now, in this case, the camera is moving, subject is moving, and you see it messes up the legs and the model loses temporal consistency. But these are difficult cases because of the amount of movements involved. Even though if there are issues, keep in mind that these videos are being generated by a single text prompt and the model is able to convert that description into pretty realistic videos. This much progress was really unimaginable just even a few months ago. So we have come a long way with these models. Before looking at some practical applications and use cases, I wanted to highlight some potential impacts that this work can have. According to the others, at the moment you can distinguish the AI generated videos from real videos, but in a near future it may not be possible to do that. Now this is going to result in some important ethical and safety implications as state of the art generative models can be used for malicious purposes. They also have concerns about the data sources that they used, and as a result, it might not be ready for commercial applications. In terms of applications and use cases, they actually showed that they are able to combine this text to video with other models. For example, Dream Booth, it's a model that is used to recreate the same subject in different scenes. So let's say you are going to have a bunch of input images. These can be animals, buildings or humans, right? And then you can recreate that person or character in different scenes. But in this case, they took it a step further. So instead of recreating the same character in a different scene, they are able to recreate whole videos based on the training data provided to a Dream Booth model. Now, for the exactly same cat, they actually recreated that cat playing in the grass based on just the images they provided. This is completely wild. Now here is the same cat just Getting up and it's based on the images it provided along with the text prompt. Now you can think about the possibilities it can have. So let's say you have a character and you just have access to the images and pictures. Now using this model you can make them do anything you want. And there are going to be some severe ethical implications of this work. Another practical use case is generating synthetic data for training, machine learning and computer vision models. So for example, since it's temporary coherent resultant videos, they were using this to generate driving videos with multiple cars. And they are realistic looking driving videos that you can actually feed into your machine learning models to train them. So this way you don't really have to drive around on real roads to collect training data for your models. These models can do it for you now and you can simply simulate the situation that you want. Closely tied use case is simulation of drivers behaviors in real world. So for example they took a simple picture and these are five different cars that are highlighted. Now using this text to video model they can simulate different behaviors of these cars. So here is one use case in which how the different cars will be driving on the road and they can simulate different situations just by using a single input image. This is an amazing work. The possibilities are endless, but it can really have some practical implications. Especially if you are able to combine it with models like Dreambooth. Now you can create realistic looking deepfakes. Let me know what you think about this work. I hope you found this video useful. If you have any questions, please put them in the comment below. Thanks for watching, see you in the next one.
