Intense AI Video Maker (Stable WarpFusion Tutorial)

Maybe you've seen some of these really cool AI generated videos. These are actually video to video where it started as a normal video and then a whole bunch of cool AI effects were applied over the top of it to give it this look. This is a video generated by Mr. Boofy here, using the source of my videos that you see here as the original video. These types of videos are really, really cool, but they're one of the more complicated styles of video to do with AI. Some other really good examples, this guy James Gerd over on Instagram has some AMAZ videos that use this style. After seeing some of his videos, he really inspired me to try to learn how to do this kind of thing. You might have even seen some of his other videos because his videos of these marble Greek statues dancing have gone fairly viral lately. Here's another great example from Enigmatic E, who also has a really great YouTube channel on AI where he shows how he actually makes this style of video. And then maybe you've seen on my Instagram or Twitter me sharing this video. This was sort of my attempt to try to do do what they were doing. Mine doesn't look nearly as clean as anybody else's that I just showed, but I've been trying to figure it out. And in this video, I'm gonna walk you through the process of how this style of video is made. Now, before we get too deep into the video, I do wanna warn you that this is not a free method. There is actually some costs involved. You're likely gonna need to have an upgraded version of Google Collab and the code to actually generate this style of video. The person who created it actually only shares it with people who are members of his Patreon. So most of the videos that I just showed you were created with something called ST Fusion. And the person who developed that is this S X E L A. I don't really know how that's pronounced. Maybe it's xla. I'm not sure. But if you support him on his Patreon for $10 per month, he gives access to his latest version of Stable Warp Fusion so that anybody can download it and use it. I'll make sure the link is in the description to where you can actually download it. It's a pretty brilliant Patreon strategy if you ask me, because he's actually getting other people like me and like Enigmatic E making videos that point people to his his Patreon. But if you join the $10 per month tier, you will get access to the download. And once you're a member. You can see in his feed, he's got Stable warp fusion version 0.16, 0.15. For this demonstration, we're actually going to use 0.15 because as of this recording, that's the most stable version from within his Patreon. We're going to download stable warp fusion version 0.15.5 IPYNB. Go ahead and click on that and we will download it to our computer here. Now, the next thing you're going to need is a model. So there's a specific style that you're going for. You're going to want to find an existing checkpoint model that's trained on that style and you can find those over at this URL here. Civitai.com this is a whole bunch of trained checkpoint models that work for Stable Diffusion, but they're also the same models that you would use for Stable Warp Fusion. Now just be warned, when you visit this Civic AI site, there is a lot of not safe for work content. I do have my filters turned on that you can see up here. If you don't have the filters turned on, you will definitely see some very adult content on here. Now the models that are on CIVIT AI are made available to download for free, so there isn't a cost to actually grabbing one of the models that you like. But let's say I want to animate something in this sort of cartoon style over here. I can click into this model checkpoint, see some examples of images that have been generated with this model, and yes, there will be some not safe for work images within here as well. But I think this would be a fun model to test inside of Stable Warp Fusion. So now we need to go ahead and download it. Please keep in mind these are very large files. This one's 5.28 gigabytes. I'm going to go ahead and click download and it's going to take us a few minutes to download this one. Now another thing that you can look up if you want to add some extra special effects to your animation is we can come back to our homepage of CIVIT AI here and we can actually come up to this filters button here and filter it by Loras. Let's turn off the checkpoints and only show Loras. Loras are sort of additional kind of add ons that you can do in combination with your main checkpoint model. So I can use that kind of cartoon animation checkpoint with a Lora and the Lora will add some additional sort of effects to the video. Again, there Are a lot of not safe for work auras, so I'm keeping them blurred out. But for example, here's one called Bubble Drip. Let's click into this and take a peek. You can see it adds these sort of cool bubble effects to images. Could be interesting to see what it does to videos. And these loras are actually quite a bit smaller than the model checkpoints, so they should download a little bit quicker. Let's go ahead and download that one. And while we're waiting for this stuff to download, it would be a good idea to pick a source video. Since most of the videos we've seen are of people dancing, I think that's a good way to demo what this is capable of. So let's find a video of somebody dancing. I have an account over at Motion Array, so I'm going to go ahead and try to find a stock video of somebody dancing. You can really get the source video from anywhere. But I'm using a stock video just to be safe to know that I have the right stock to use this video. So here's an interesting video that will probably look fairly interesting once it's ready. So let's go ahead and use this one. Now, the video is 207 megabytes, and it's an MOV file. I'm actually going to shrink it down a little bit, make it smaller of a video file just because it'll be easier to work with. So to do that, I'm going to quickly pull this video into DaVinci Resolve. The whole thing is 24 seconds, I think. I want it to just be about 10 seconds. So I'm going to cut off a little off the frame front. So I got it down to about 12 seconds. This is the video that I'm going to work with here. They're doing their little dance, and they kind of all come together. They do this little, like, wave motion, and then it ends right there. So I think that's a pretty good little clip to use. I'm going to render this out in MP4. This will shrink it down so it won't be nearly as large of a file size. And just to make sure the process is fast and smooth, I'm actually going to bring the resolution down to 1280 by 720. And we'll render it out like that. Now, if I show you this file, you can see it's now down to four megabytes instead of 200 megabytes. All right, so we've got our checkpoint. We've got a lora that we can add into the mix. We've got our notebook file that we can use with Google Collab. Now it's time to open up Google Collab and you can find Google Collab over@collab.research.google.com and when you first go to this URL, you'll see this screen here. What you want to do is you want to click on the upload button over here and this Stable Warp Fusion IP YMB file that we grabbed earlier, we're going to take that file and we're going to drag and drop it right in here. And now this is what Stable Warp Fusion looks like. Now, I'm not 100% positive, but I'm fairly sure that the amount of compute power that this takes requires a Google Collab Pro account. So you can see I am on the Pro account. It is 9.99per month. So in order to follow this Stable Warp Fusion process, you do need to pay $10 on Patreon to support the creator of Stable Warp Fusion to get the downloadable notebook file. Also, you're probably going to need the Google Collab Pro plan, which is also 10 bucks a month. So just keep that in mind. You kind of really gotta want to do a lot of these to make it worth your cost. All right, now that we're in the Google Collab notebook, it's going to look overwhelming. There is a lot of settings that you can adjust and tweak here, but luckily there's only a few that you really need to mess with. So I'm going to walk you through them real quick. So to start, come up to runtime up in the top menu and click Change Runtime Type, set the GPU type to A100 and then go ahead and click Save. This will make sure that you're running a higher end GPU that could handle the processing of Stable Warp Fusion here. Then we're going to come up to the top right where it says connect and we're going to go ahead and click that and it's going to set up our gpu, our cloud GPU that is using behind the scenes. So we're now connected and now we're going to scroll down a little bit. We can scroll past everything here, past the change log the credits, and then you can see we have a section called Setup. Now, all of the files that we're going to use, the Checkpoint file that we downloaded, the Lora file that we downloaded, the video that we're going to have it process, all of that is going to run from within Google Drive So we need to go ahead and let it connect to our Google Drive. So we're going to press Play underneath this Prepare Folder section and then go ahead and click Connect to Google Drive. Go ahead and connect it up and click Allow. And now we've got Google Drive turned on. Now if I open this little folder on the side that says Files, you should see a folder that says Drive. If we open this up, we now have access to our Google Drive here in the back end. Now we don't have to worry about any of these other Play buttons yet. There's a lot of steps here that we can just kind of skip. The next thing we're going to do is under Settings where it says two. And then Settings is we're going to change this batch name. And this is going to be the name that is going to give the output file once it's done processing. So let's go ahead and do animated dancers. And then we need to make sure that the width and height is set to that of our video. This one's currently set for a vertical view. The video that I'm doing is actually flipped. It's actually 1280 by 720. I'm going to go ahead and flip that here. But just make sure it's set to the aspect ratio of the video that you're actually working with. We can skip past these settings here. And then under Video Input Settings, it asks for this video init path. This is actually where the video is going to be located that we're going to have it process. So I created a folder inside of my Google Drive called Warp Fusion. I'm going to go ahead and open this folder here. And then this little future dance video that I created, I'm just going to upload it into this folder. All I have to do is drag it and drop it right into this Warp Fusion folder. It's going to take a second here. And now you can see it is uploaded inside this folder. If I click these three dots here, I can click Copy path. And then next to Video initial path here, I'm just going to replace this. I'm going to click Paste. And you can see now I've got the path to that file that I just uploaded. Now this extract in frame here, you can actually make your video process faster by changing this to two if you'd like. If you set it to two, then what it's going to do is it's going to process every other frame instead of every single frame. Now this is sort of a personal choice. If you have it process every other frame. It will do the processing quicker, but it'll probably look a little bit choppier if you do every one frame. It's going to process every single frame of your video. So it'll look a little bit smoother in the final output, but it will also take a little bit longer. Since I'm only doing a 12 second video here, I'm going to extract every one frame. It's probably going to take, you know, a good hour to process, but I could live with that for this video. So we're going to scroll past all of these settings down here. It says store frames on Google Drive. I'm going to go ahead and click that so that every single individual frame that it creates, it's going to store each individual image inside of Google Drive. Again, there are a lot of settings here that if you are going to use Warp Fusion, you can, you can definitely play with them and test different stuff. I'm just trying to make a real basic, simple video for this tutorial. So all of this stuff you can play with extract background mask. It will try to rotoscope the main character that you're trying to generate and create a mask from that. I'm not going to bother with that. For this tutorial we're going to scroll down to this section that says define SD plus K functions. Load model. Under load two, I'm going to change this from CPU to gpu. And then under model path, this is where we're going to put that checkpoint that we downloaded from Civit AI. So under this Warp Fusion folder that I created in my Google Google Drive, I'm going to open my Models folder here. You can see I have a handful of models already in here, but I don't have this Yiffy Mix one that we just downloaded. And I can't really just drag and drop it over into this browser here. It's a little bit too large to do that. So what I need to do is actually open up Google Drive from within my computer. So I'm going to go ahead and copy this file here. I'm going to jump into my Google Drive, go to my Warp Fusion folder that I created, go into my Models folder and I'm going to paste it using the File Explorer on my computer here. So now it's in there, but it still needs to upload to Google Drive. You can see it says upload queued, upload 0%. So it's going to take a little bit before this model is fully inside of my Google Drive and Usable. While we're at it, might as well do the same thing with our Lora file. So I grabbed this SY3 safe tensors, that's the Lora file that we grabbed, that adds those little, you know, bubble drips. I'm going to copy this file here. Once again. We'll jump to our Google Drive, our Warp Fusion folder that we created, and then I have a Lora folder. I'm going to go ahead and paste that into the Lora folder here. And once again, checking Google Drive, you can see it's going to take a little bit of time for that to process. So while that's processing, we can move on to some of the next steps here. It's asking for a control. Net Models directory. So under this Warp Fusion folder, I also created another folder called control. Net. You would do that by just going to the Warp Fusion folder over here that we created earlier, clicking on the three dots, clicking new folder, and then just creating one called control. Net. I've already done that. I've got my folder control. Net here, so I'm just going to copy the path. And then in my control. Net Models directory, I'm just going to go ahead and replace that with the path that I've got there. Once my checkpoint file is done uploading, I will replace that URL in a minute. So we'll jump back to that in a second. We can scroll down to our Lora and embedding paths here. We already created our Lora directory earlier, so I'm going to go ahead and click on these three dots, copy the path, and make sure our Lora directory is set properly. I'm not using any custom embeds for this one, so I'm just going to leave that alone for right now. Okay. So our model's finally been uploaded. You can see it under our Models folder here under yiphy mix. So I'm going to click these three dots copy path and then under define SD +k functions next to model path. I'm going to go ahead and paste this new model that we just downloaded from Civit AI here. Now we are actually pretty much ready to run this. There's one last step we want to do. We want to kind of give it a prompt to start. So what we'll do is we'll scroll down until we come to this one that says non GUI here. You can see we've got some cells hidden. So we'll go ahead and view the cells that are hidden. And if we scroll down a little more, you can see you can Set a seed here. This can be any random number you want. And then down here you can see this is where you would enter a prompt. If you don't enter anything here, it's by default just going to use this prompt that's already there. A Disney style cartoon animation of robots dancing. Colorful, highly detailed. Let's throw Pixar in there. And then we'll use that as our starting prompt. Now if we want to use that Lora, remember we looked at this bubble drip Lora here. There's a little bit of a code that you have to use to make it work. The easy, easiest way to learn how to use that is to come down to some of these examples of what other people have done. You know, click on the little info button on one of these images and see how they use the prompt here. This prompt uses this little code SY3. So we can actually just copy this right here, bring it over to our prompt and add it to the end of our prompt here. Paste that in. It uses these little brackets. Lora colon SY3 colon 0.5. This is a weighting. So if I give it a weight of one, it will use that. Lora a little bit heavier. If I give it a lower weight, it'll be a little more subtle. Let's go ahead and leave this one on 0.5 to get a similar look to what we see here. We're kind of blending some styles here, so I have no idea how this is going to turn out. We'll leave our negative prompts the same that they already have here. Maybe I'll add disfigured, ugly, mutant, some stuff like that. That's going to possibly make it look a little funky. And now all we have to do is come up to the top here, come to runtime and just click run all. And this is going to take some time. We're going to watch it run all of these processes step by step. Some of them are going to take a few seconds. This one only took four seconds. Some of them are going to take a few minutes. As we move down a little bit, you can start to see it processing some of the control nets. So you get a little idea of how it's outlining and seeing the original images. Okay, so now we're finally to the part where it's running the portion that says do the run. And what's really cool about this is in a minute here, it's going to show us a frame. It's going to give us a sort of example of what things are going to look like. And if we don't really like the way it looks, we can come back up to this new graphical user interface that it created for us here and tweak our prompts and tweak some of our settings. So let's go ahead and scroll back down and you can see now it gave us a sample of what our first frame looks like. So let's say I'm not really super happy with the way this is coming out. I can go ahead and click Stop right under do the run here. Now we can scroll back up here and under our new user interface, we have a bunch of settings that we can start to play with. Let's change this to a Pixar animation of robots dancing. Let's get rid of colorful. Let's make it in space, stars in the background. And let's go ahead and up our Lora effect up to 0.9. We can also click on our Control. Net tab up here and change which Control. Net models it's using. I'm going to go ahead and turn on depth and turn off the inpaint and there's just a ton of settings that you can play with to get what you're looking for. Again, since this is just a demo to show you how to do what we're doing, I'm not going to play with every single step. This is for you to get in and play around with. Once I've changed a few things and I'm ready to go again, I'm going to come down to under Reference, Control. Net, Click Play again, come down to diffuse. Click Play again on that one. And then once again, we're going to click Play on, Do the run. We'll wait a minute or so and it's going to give us another preview of what this new version is going to look like. So here's our new version. I like this version a little bit better. You can see it's actually adding some more of these little bubbles from our Lora, using those as kind of the stars. So I think this is a better version and I'm just going to let it run now. You can see it's 1%. It's estimating it's going to take roughly an hour, and I. I think 41 minutes or so to run this. So at this point, just let it run. We're gonna step away from the computer and just let this thing process for a bit. All right, so it's finished. Processing the run, you can see it took pretty much about two hours. Exactly. Here's the final Screenshot, it moved onto the process of creating the video. And what it did is if I open my Google Drive now, so I'm gonna go ahead and just open it on my desktop. It actually created a folder inside of my Google Drive called AI. And if I just double click into this AI, there's a folder in there now called Stable Warp Fusion. This was all automatically created. There's a new folder called Images Out. And because I've done some other videos in Stable Warp Fusion, there's other folders in here, but this is the Animated dancers one that we just created here. So if I double click in here, you can see every single frame from the video is loaded in here. So you can actually click through them and see them all. But what we really want to know is the video. So if I double click on this folder that it created called Video, it actually rendered it all into a single video, and we can watch the entire animation just like this. We've got our dancing robots moving around, doing their little thing. The stars in the background, the spaceships in the background. Now it is pretty flickery. So one way you could sort of fix that is we can actually open up DaVinci Resolve, drop our video into the timeline here. Go ahead and let it use the frame rate from the video. You can see, here's our whole video. If I come over to the Fusion tab here, you've got this line that's got Media in and Media out. I click on this line here and hit Control space bar. It brings up this ability to select other tools. And then I can type in deflicker and bring in the D Flicker tool. Click Add. And now it added the D Flicker. Now if I come up to the D Flicker settings, we've got some options here for time lapse or fluoro light. If I select fluoro light, come back to the timeline and watch the video, it actually removes some of the flickering. It's not perfect. And then if we want to do kind of a comparison shot, I could pull this original video into the timeline. So you can see it here. It should line up pretty perfectly with our existing video. And then if I want to add that wipe effect, I can essentially add a keyframe. So let's come a few seconds in like this. Let's add a keyframe on our cropping. I'll open up our crop down here. We'll click on this little diamond to add a keyframe for our crop. And then if we slide a few frames forward, add another keyframe, and then bring our crop right all the Way across like this. Now it will animate between them with that little sort of wipe effect. And then if I want to do that again, I can come up here, click our little diamond, move a few frames forward, click the diamond again, and bring our crop all the way back. And now we've got this effect here. And then in a minute it's going to zoom back. A little extra trick you can do if you want, but let's go ahead and get rid of this one. Render this out with the extra D flicker and see how it compares. Here's the original version before adding D flicker. Now here's the final product. It does still have quite a bit of flicker to it, if I'm honest. But I think that's only due to my own lack of knowledge on all of the various settings inside of Stable Warp Fusion. There's probably some settings that I could have tweaked. Tweaked. I just am still learning Stable Warp Fusion myself, and I really think I probably could get it looking a little bit better with some more time. So I'm looking forward to seeing what other people who watch this tutorial go and do with it. Now, just for fun, I also ran that same video through Kyber. I gave it the prompt of futuristic robot dancers that are dancing in space surrounded by stars in the style of Pixar animated cartoon. And this is what it gave us. Here's the larger screen version of the animation. It's definitely a lot less flickery than what I got out of Stable Warp Fusion. But also it doesn't quite have the same coolness factor to it doesn't have the same level of contrast and chaos that was going both. Totally different styles of video, but both pretty cool in their own right. Me personally, I'm probably gonna use Kyber more often than I use Stable Warp Fusion just because of the process being so simple and so quick. But you definitely can get it much more dialed in and much more exactly what you're looking for using something like Stable Work Fusion. Because there's all sorts of different models that you can use from CIVIT AI. There's all sorts of different Loras that you can use to add little extra effects to it. There are so many different variations and really so many different ways you can get it dialed in. With Stable Warp Fusion. It's kind of like the difference between using automatic 1111 and stable diffusion to get an image and get that image exactly what you're looking for through all of the little buttons and knobs and little settings that you could tweak versus mid journey where you have a lot less options. You can kind of prompt something and you sort of get what you get and it looks cool, right? So Kyber is sort of like the mid journey of creating this kind of videos where you're going to get what you get out of it. Stable Warp Fusion. Consider that more like the Stable diffusion, the automatic 1111 where you can really, really get it dialed in and get way closer to the exact, exact video you're looking for. So anybody who's been interested in Stable Warp Fusion, that's. That's my tutorial on it. I know this is probably not going to be one of my more popular videos on YouTube. This is a very niche one that only a handful of people probably really care about watching. But it's one that I wanted to make because I'm learning Stable Warp Fusion. I want to get better at it. So I wanted to show the process. I want to show where I'm at with it right now. I Also see my YouTube channel is a little bit of a catalog of various tutorials along with the news. And a Stable Warp Fusion tutorial was something that I just kind of wanted in my YouTube catalog. Once I get better at it, I'll probably revisit it and show you even more cool tips and tricks that I learned as I get better with it myself. Thanks so much for tuning in. I hope you enjoyed this video. If you enjoy tutorials like this, give this video a thumbs up and I'll make sure you see more videos like it in your YouTube feed. Also, don't forget to check out Future Tools where I show off all the latest cool AI tools, all the latest AI news and join the free newsletter. I'll keep you in the loop of all the latest AI news and tools and I'll send it every single Friday. You can find all that over at FutureTools IO. Thanks again for tuning in. Really, really appreciate you. I had fun with this one. Hopefully you enjoyed watching it. See you guys later. Bye Bye.