LARRUZ walks through Runway ML's Gen 2 text-to-video tool and two free ways to use it: a Hugging Face web demo and a Google Colab notebook. Gen 2 can turn text prompts into 1-3 second videos, plus do image-to-video and style transfer. Outputs come with Shutterstock watermarks, which you can strip with Any Eraser.
Questions this video answers
Tesla T4
Around 5 gigabytes and another ~4 gigabytes
15th
Any Eraser
Uploaded Mar 21, 2023; creator: LARRUZ
Chapters
If you thought text image was cool, wait till you see text videos and generate two to three second videos just by using simple text prompts. And these videos are temporally coherent. So here are a few examples that people have been playing around with. The best part is the model is open source and you can play with it in your own browser. In this video, I'm going to show you how to generate these cute little videos yourself. Before showing you how to run these text to video models, I want to show you this gen 2 models from Runway ML. They recently announced it, it hasn't been released, but it has very similar capabilities to text to videos. So for example, in this case you can provide a simple prompt of aerial drone footage of a mountain. And this model will be able to generate a video of a drone going through a range of mountains. So it's another example of text video. It actually synthesize the videos based on the style that you mentioned. Then you can combine actually text plus image to video. So for example, you can provide an input video, then a text prompt. So here's the text prompt and based on this, this will be able to output a small 1 to 2 second video. Here is an example of another capability. So you can provide simple image and it can generate a video based on that input image. Here's another feature. So you can provide a video and then a driving image and it will change the style of the video based on the driving video. This is pretty cool because there are no flickerings and anything. So it's really amazing. Now they have some other capabilities as well. For example the storyboard. So you have a mockup and it converted to a fully stylized animated render. So enough with the Runway ML Gen 2, let's look at how you can generate videos from simple text prompt. So right now there are two ways of using this text to video synthesis model for free if you don't want to run it on your computer using automatic 11. So one is using this hugging face spaces. They have made it available for people. And the second one is through Google Collab. So I'm going to show you both of them. I will be creating a video on how to run this in automatic 1111. So if you're interested in that, just keep an eye out for that video. Okay, so before generating our own video, let's look at a few examples. So for example, they have provided this prompt, an astronaut riding a horse. So you can see that there is a horse, an astronaut and the horse is running around. So let's look at another example, a panda Eating bamboo on a rock. Right? So there's a rock panda and it seems to be eating a bamboo. Right. And Spider man is surfing. It's actually pretty cool. You see this one? Now these videos, if you watch them, they're temporarily coherent, right? But one thing you might notice is there is this Shutterstock text on it. I'm going to show you how to actually remove this from the generated videos. The quality is not great, but keep in mind it's just the first iteration of the model. So just give it a month or two and there is going to be a drastic improvement in terms of the quality. Now let's give it our own text. So let's say a dancing skeleton. Okay. This is going to take some time. As you can see, something went wrong. The application is too busy. Keep trying. So you can keep trying. And a lot of people are currently using this, so expect that it's going to take a while. So now if you notice I am 15th on the queue, it will generate the video once my number comes in. If you want faster inference, you can actually pay for a GPU on hugging spaces and then you will have a dedicated GPU that you can use. But right now I think I'll have to wait a little bit for this to complete the processing. So it took a little while. But here's the video if you run it. Yeah. So there's a skeleton which is dancing. This is pretty cool. Okay, so it's one option. The second option is you can actually use Google Colab. All right, so here's the Google Colab that we are going to be using. I'm going to put a link in the description of the video. So first you want to make sure that you actually come here and create a copy of this. So simply click on save a copy in the drive. It will create a copy of the Google Colab. Then you want to make sure that you are running it on a gpu. Right? So change the runtime to GPU and then simply come here and click Connect. If you haven't used Google Colab before, you can watch my videos where I go in details on how to use it. Next you simply want to come and click these play buttons. So this will execute a cell, for example, this Nvidia smi. So it simply tells us that there is a Tesla T4 GPU that is being used in this case. Right. So I ran this cell, then ran this cell. So it downloaded the repo in the model and installed all the required packages. Right. And currently I'm running this cell so if you see this test underscore text, this is where your prompt is going to come in. So this is the only thing that you need to change. So let's say account surfing on the waves in the ocean, you can give it some other prompt and then simply wait for it. It will take some time because it has to download a couple of models. So for example, this file is around 5 gigabytes and then it is down currently downloading a 4 gigabytes file. So that why, that's why it takes some times, but it's better than the hugging face because in that case you are in a queue waiting for your turn. But in this case at least you have a dedicated Google Colab instance that is running your model. Once the execution of this cell is complete, we will simply run the final cell where we will be able to download our video. So while we are waiting for this, let's look at a couple of other examples. If you look at all these generated videos, you will notice this. There is this shutterstock logo on all of them. And it's probably because that the videos that were used for training the model are from this website. But there is an easier fix to actually get rid of the the logo. So let me show you how to do that. So for this purpose we are going to be using this website called Any Eraser. All you have to do is go and create an account, right? And log in using that account. Then you need to upload your video. So come here. Click upload. Right? So for example, I have this video of a cat running in a playground. Okay. In order to remove this logo, come to this AI model and then it will let you select size of your brush. So for example, I will make the brush a little smaller and then carefully select the logo or area you want to remove from your video. Right? So for example, here we go. I will try my best not to select the cat. Okay, I think this should be good. Right? Then simply click on remove objects now and let it process so the process is complete. Click on download. Okay, so here's the video. I think it did a pretty decent job. Probably I missed something a little bit here, but overall it did a pretty good job of removing the logo. Now one thing you need to be careful about it is if you want to use this tool for multiple videos, then you have to pay for it. But there is actually a way around it as well. All you have to do is just erase your cookies and you will be able to upload another video and run it through that. So there you have it a quick video on how to generate videos from simple text prompts. These are really exciting times. Just give it a month or two and we will be able to generate much better quality images. Thanks for watching. See you in the next one.
