The Best AI Video You Can Run Locally

What if I told you these videos you are seeing were generated by an AI model that you can run locally on consumer grade hardware. And these are probably one of the best results that I have seen from a local model. This open weight model is from a company called Light Trex who are also the sponsor of today's video. But the model itself is pretty impressive. It's called LTX Video. The model is extremely fast. According to LitRex, it's up to 30 times quicker than any other model. And the clips actually looks pretty good. For a local model, you can render in full hd, upscale in a single click and even stair the animation with Keyframe references. Here's the official GitHub repo of LTX video. The model weights are already available on Hugging Face. So if you want to run this locally, you can download models from Hugging Face and you'll be able to use either Comfy UI or the diffusion package to run this model. The results that you get are really impressive for a local model. So this is a new 13 billion model, but there is also a quantized version which is going to need a lot less vram. In fact they are saying that you will be able to run this quantized version on a 4090 or 1590 consumer GPU. There are also two new upscalers which will improve the quality of output video. And also there's better prompt adherence and physical understanding of objects. One of the key innovation is their new pipeline for multiscale video rendering. So essentially they first generate a low resolution video in and then progressively add more and more details. This enable very fast generation and high quality results. To run this model locally you can use Comfy UI or use the diffuser package. However, I'm going to show you a much easier way if you just want to test out the model. Okay, so we're going to be looking at LTX Studio which is a platform created by Light tricks. Here you have the same 13 billion parameter model available along with a 2 billion model. LTX Studio is a more comprehensive platform that enables you to do storyboards motion generation. This is the video generator, Image generator and recently they also added VO2. I am going to be creating a more dedicated video on this, but let me show you how to get started. Okay, so we're going to click on motion generation. Now Here you have three different options. The 13 billion model, 2 billion and VO2 from Google. You can add an image For Keyframe, you can add start and end frame and it will generate motion between them. And also you can automatically generate sound effects. Based on the video, I'll show you a couple of really cool demos. Let me first show you some of the features, and then we're going to look at more complex examples. To start off, we're going to use this image without any text prompt. And let's see what the model can deduce from this image and what type of generation we get. Okay, so I uploaded the video. I'm not going to add any end frame for this video. Now you get four different options starting from three seconds all the way up to nine seconds. Let's keep it to seven seconds. And then you also have the ability to control how much motion is going to be introduced by the model. For this example, we're not going to provide any text prompt. I just want to see how good the model is at understanding videos from a single frame. Okay, so here are the results, and I think most of them look pretty good. They are able to introduce motion, which we would expect based on the image that we saw. Now, I personally like image number or video number two, because this looks a lot more realistic. Although the amount of motion in this specific video is not that much. Now, for some of the other. These other ones, for example, here, it seems like it changed the scene. So instead of that downtown background, it seems to have changed that. But overall, all of them look pretty good. Now let's Change this to 100% and let's see what the results actually looks like. So it's going to be using the same input image. Okay, so you can see there is a lot more motion now. But keep in mind, this was without any text prompt. Let's add a text prompt and see what we get. Okay, so here's a very detailed text prompt that describes the camera motions, how the car drifts, and so on. So I'm going to still keep it to 100% motion intensity. And let's regenerate this. And after that, I'll show you a couple of other cool things that you can do with the generated video. Okay, so here are the results with 100% motion intensity. And I reduced it to 50%, I think. I personally like the 50% motion intensity. The results are pretty good for a 13 billion model. If you look at these videos, it is actually able to preserve the shape of the car, which is pretty neat. But let me show you a very cool feature. So let's say we select one of the videos. This is the video that I selected. Now, what I can do is either I can upscale this video or I can add sound to it. So There is this auto SFX option. If you click on this, it is going to look at the video and based on the video it will automatically generate a sound for it. Now this is pretty neat feature and look at the quality of the sound output. And this is pretty awesome because you can actually see that it looking at scene and then based on that it's adding sound effects. It's one of my favorite features of LTX Studio. Okay, so there are some other really neat features within LTX Studio. You can upscale your videos and images. And then there are these two options. Generative fill and remove objects. Let's say we want to remove this car from from the video that are going to be generated. So I'm going to just select that car. You can also add objects by using the generative fill option. But let's see what it does. So let's click on that and then we're going to just simply ask it to generate a video by removing those objects or that specific car in this case. Okay, so now it generated four different videos without that car in the beginning. So essentially we removed the object from the image before feeding it into the video generation model. Now for some reason it decided to not actually generate a video for the first one. But the good thing is you get four different video generation so you can actually choose which one works the best. Before showing you some more advanced features within the platform, I want to show you a few more examples. We are going to also look at some failure cases in which the model is not able to do that. Great. But before everything else, I want to show you this video. This was I think one of the first open weight AI video generation model a couple of years ago. And it really showcased how far we have come. Video generation is particularly difficult because the model has to produce frames which have temporal coherence. And I'm always fascinated when I see videos like this in which the object rotates but still it actually preserve the shape. Okay, so this was a quick test of sand flowing through somebody's hand and it does a really good job at it. Now there are definitely some weird artifacts. For example, if you look at this video at the end, the sand kind of goes into the hand, but overall the results are pretty good. Here's another test. In this case, I provided an image that has some text and not only it produces pretty realistic motions, but it also was able to preserve the text. So this is pretty neat. Now one thing about LTX Studio or this model is that you can provide a reference image which I think is a really good feature because this gives you a lot more control on what exactly you you want your start frame or end frame to look like. If you just use text to image or text to video models, you don't really have control on the content. Okay, so here's another one in which I asked it to create a video of an athlete who stops after jogging and then he's drinking water. The results are really good for 13 billion model. Okay, so it had actually a lot of struggle with this particular video. So in this case, I asked it to create a video of a Mercedes Benz that stops and then all of the four doors opens. You can see some weird artifacts. It seems for these couple of videos, it's basically opening up the wings rather than the doors. This one still look relatively realistic. And in this case it didn't even open the door. But I think one of the most powerful use case is for products. Because in this case I provided a reference image of this handbag and then I told it the type of movements that I want. If you look at this, it actually preserves the shape of the handbag throughout the video, which is pretty incredible, particularly for this one, actually deteriorates the shape and you don't really see some weird artifacts. Right. In this case, I think the bag handle is not complete. Right. But the good thing is you get to generate four different videos. And if you don't really like a video, you can actually regenerate the same video again. Okay, so here's one of skiing. You can actually see that it produces realistic physics because as the person is skiing down the mountain slope, you can see this trail of ice or snow, but it's very realistic. But there are some cases in which the model is not able to figure out what exactly to do. So for example, here was a prompt I wanted for the days to transition into night. And then I also wanted to have magical solar eclipse. Right. Now in this case, what I did was I had the image as a start image. So the model is not actually able to figure out what exactly to do with it. Because the start image is essentially a night shone. When I replace it to the end frame, I think it produces visually better results because it actually transition the day into the night. Although it came up with very interesting interpretation of the image itself. So this is where the prompt comes into play. You need to provide very detailed prompt of what exactly you want. Also, within nine second video, you, you cannot do a lot. So you need to be very careful of how much movements and motions you want. Now in cases, if there are Too much motions. You're going to see some very interesting results. So this was based on an AI generated image, and I wanted it to produce a chaotic food fight. And this is what it comes up with. Actually had a lot of fun. Okay, next I'm going to show you how you can use the keyframes to control movements. But instead of defining multiple keyframes for the same scene, I want to see how it's going to use keyframes to transition between these two images. So this is going to be the start image, and here's going to be the end image. And let's see how it comes up with the transition. I uploaded both the images and in the beginning, we're not going to use any text prompt and let the model decide the transition between these two images as keyframes. Okay, so here are the results. Keep in mind, this was without any text prompt. So the model has to come with the transitions. And I think it did a really good job, especially for number two and number three. In number two, it transforms that aircraft into a robot. For number three, it is generating a camera movements around the aircraft. And then in the background we have a robot for one and four, I think it does try to transform the aircraft into a robot. But I would personally say I like number two and three a lot more than one and four. Now for these four, I did provide a prompt which was basically transform this jet aircraft into a robot. And I like number four because I think it did a really good job. Here, keep in mind, is a very complex transition. 13 billion model is still able to hold up pretty nicely. I think the keyframe can be really great for making product videos. For example, I created this transition video based on the first and the last frame. I think it turned out pretty decently. Okay, this is also great for some sort of short or vertical videos. So let's just going to select that. Okay, I'm going to add this video. There's a dragon in the background and a very simple prompt. Let's see what it comes up with. Okay, so the first set of videos that you see there are without any text prompt. And it figured out what a user would expect, especially I guess, like this one. If you look at this video, the dragon is coming towards both of these characters and then flying away. This is probably kind of effect that you would expect based on the image that we saw. Anyone? I think the actual scene was something like this. Now, for the second set of videos, I did add a simple prompt which was basically a dragon flying towards both the characters. But still, the video quality is pretty nice based on only a 13 billion model. Now I do want to see what type of sound effect the model is going to add. So let's pick one video maybe. I'm trying to decide which one to use. Okay, so I'm going to pick this one and let's see what type of sound effect it's going to use for this specific video. All right, so here's what it added. Okay, this is probably a hard one to do, but still pretty impressive that it added the sound of the wings. Another interesting feature is the ability to add camera movements. So let's click on camera control. We're going to upload an image. So here's an image of a robot. Then we can add custom camera movements. So either you can use your mouse to add different movements, or you can click on some pre select movements like this. And you can also switch between whether this should be the start frame or the original frame. So let's see what this movement is going to look like. As you can see, it has to fill the background as well. So let's apply this. I'm going to just keep it to three seconds. Let's hit generate. Now here's the output. It really gives you finite control on what type of movements you want to introduce in your shot. Okay, just for fun, I added audio SFX to it and here's how it sounds. Okay, here's how an upscaled version of the video looks like. So the quality is definitely much better compared to what we saw before. Keep in mind, upscaled version is a paid feature, but you can try out the platform for absolutely free. The type of results that you can get from this model are really impressive. Get given its size and also the ability that this model can be run locally. Now, the quality of output that you're going to get is going to be dependent on the quality of your prompt. So make sure you add a lot of relevant details and frame the shot as you would want. Don't let the model make any assumptions. Anyways, do check it out there. This is a very impressive model for its size and you can use the LTX studio for free. Let me know what you think about this model and what type of generations you are able to produce. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.