Actual AI Text-To-Video is Finally Here!

So up until now, we haven't really seen real text to video. We've seen some demos from companies like Meta and companies like Google showing off what text to video is coming. And we've had some really cool tools like Deforum and this plasma punk tool and this decoherence tool which sort of merge image to image to image and give kind of a cool animation effect. But we haven't really had true text to image in the way we would use Stable Diffusion or Mid Journey to type in what we want to see and actually get a video of that thing until now. In fact, here's a cool demo of what some people have done that I came across on Reddit. You got mountains and water in a Chinese painting, a beautiful painting of a Buddhist temple in a serene landscape. Traditional Chinese painting, landscape with bridge and waterfall. You got fireworks. You got campfire at night in a snowy forest with starry sky in the background. You've got a mountain river. So I'm gonna go ahead and play this so you can see what all these various demos look like. And then we're gonna go play around with it ourself because you can use it right now today. You can see the mountain and water. You can see all these waterfalls. You can see the fireworks. Here's the mountain river. The starry night with the fire going. We've got a clownfish swimming through the coral reef. We got ducks swimming in a pond. A litter of puppies running through the yard. A panda bear is eating bamboo on a rock. A horse is chewing. A knight riding on a horse. Animation. All of these were done with text to image. An orange cat wearing a leather jacket and sunglasses sings in a metal band on stage. A monkey learning to play the piano. Two kangaroos are busy cooking dinner in a kitchen. This is from a Reddit post that I found in the Stable Diffusion subreddit here, called the first open source text to video. 1.7 billion parameter diffusion model is out. Now you can play with this right now in this hugging face space here called Model Scope Text to video synthesis. Now there is kind of one little catch to it. It seems as though a lot of the videos that are inside of this model that it was trained on were actually videos that seem to have been taken from Shutterstock. So a lot of the videos that it produce has Shutterstock watermarks across it. For example, Victor M here on Twitter, who is the head of product design at Hugging Face, had this post go somewhat viral where he generated his own little Star wars clip using AI. Now, if we watch this Demo video that they made. You'll notice kind of across the bottom of the video there's sort of a Shutterstock watermark that kind of shows up throughout the whole thing, which I think proves that a lot of the video was trained on Shutterst.

If you want to play with this yourself, here's what you can do. You can go to the hugging face model, scope text to video synthesis. I'll make sure it's linked below this video here. And there's really two ways that you can do this. You can do it for free right now in hugging space, where we enter a prompt. And it will probably take a little bit because a lot of people are playing with this right now. So let's do like an alien eating a taco. And if I click run, it says, this application is too busy, keep trying. Let's go ahead and try it again. Application too busy, keep trying. So you might be able to run it after some time, but a lot of people are messing with this right now because this is fresh, this is new. This is the hottest thing at the moment. However, you can duplicate the space yourself, but you will need to have a credit card on file inside of hugging face. And it will probably cost you a few cents, like probably less than $2 or something to run it. So for the sake of this example, I'm going to duplicate the space so that you can see what it does. But keep in mind that if you really want to do it for free, you could. You just got to keep on trying while their servers are really bogged down. Let's go ahead and duplicate this space here. And it's going to take a little bit of time to get the space up and running. Okay, so I've duplicated the space, but it's showing me that I've got a runtime error. And the reason for that is because it duplicated it onto a flat free GPU which isn't going to be powerful enough to actually run this model. So if I come over here to Settings, you can see it's got it on this CPU basic, and we're going to want to do something a little bit stronger than that. So I'm actually going to upgrade it to this T4 medium here with 30 gigabytes of RAM and that should be enough to actually run this. So let's go ahead and swap to this. I'm going to need to add my payment method in here and I'm going to set a sleep timer of just one hour of inactivity. So if I accidentally walk away it doesn't continue to charge me. So we'll go ahead and click confirm new hardware here. And now it's going to try to boot up this T4 medium and that should get rid of our runtime error once that's all booted up. All right, and now we are running on our own T4 system, so I should be able to generate whatever I want and not have to wait in any sort of queues. And let's go ahead and try a green alien eating a taco. And let's click run. You can see it's actually processing. This time I'm not getting any errors because I am not having to deal with everybody else using the exact same server as me. I've got my own server now and our video is ready. It took about 60 seconds to process. Now it is only a two second clip and you can kind of see a little bit of a watermark running through it. But here's our green alien eating a taco. Maybe I can see a taco in there. Maybe if we give it some more detail here, if we do the standard thing that you might do on like stable diffusion, a detailed green alien style standing on a red Mars landscape. Let's add some of these other keywords like unreal engine trending on art station. Realistic. Realistic as an alien can be, I guess. HD4K. Let's see a detailed green alien standing on a red Mars landscape eating a yellow crunchy taco. So now we've got a little more detail. Let's see what happens when we run it this time. All right, it looks like we have a little more detail. I'm not seeing a taco yet, but let's go ahead and see what we get.

Well, I'm getting my alien on Mars, but I'm not really seeing the taco part. We're still getting that Shutterstock watermark that seems to be on every single video, which I think just sort of proves that all of the training material that they use was probably Shutterstock videos. Let's try a different subject matter here. Maybe something a little more realistic. Let's do a penguin kicking a soccer ball. All right, I'm seeing a soccer field here.

Oh, you see like a penguin flash on the screen real quick and then fly away. So far I'm not able to get anything nearly as detailed as what we're seeing in some of these videos here. Let's try some that were actually in this demo and see if I can get a similar result from the demo. Like clownfish swimming through a coral Reef. Let's see if we can get something similar there. All right, so we got clownfish swimming through a coral reef. Let's go ahead and run that and see what we get. All right, this is looking a little bit better now. You can tell that this is supposed to be a clown fish swimming through a coral reef. Let's try a monkey on roller skates. All right, I see the monkey. I'm not seeing roller skates. Oh, okay, okay, okay. Yeah, that's a monkey on roller skates. Sort of towards the end of the video, you can see it. I really wish you could generate longer than two seconds, but, you know, it is what it is right now. This is obviously very, very early tech. Let's try a cat learning to play the piano. All right, let's watch our cat playing the piano here. Kind of more Looks like a cat sniffing a piano. Now here's the model scope page for this text generation video mock up. And you can see some of the ones that they shared. Giraffe underneath a microwave. It kind of looks like a giraffe in a microwave. A golden doodle playing in a park by a lake. A panda bear driving a car. Teddy bear running in New York City. Drone fly through of a fast food restaurant on a dystopian alien planet. A dog wearing a superhero outfit with red cape flying through the sky. Now I gotta be honest, I think these are some of the cherry picked ones. Cause when I try stuff, I'm not getting the greatest results. So I think, you know, they probably did a thousand generations. And they're showing you the nine best ones that they came up with. Cause so far it's, you know, not that close to what I'm trying to generate. Let's go ahead and try a dog wearing a superhero outfit with red cape flying through the sky. See if I get something similar. See, when I use that exact same prompt, that's what I get. It's like a dog with maybe a cape wrapped around it. And that's the exact same prompt that they're showing here is a dog wearing a superhero outfit with red cape flying through the sky. And I'm sure I can use a different seed and get a completely different output. So let's go ahead and change the seed, get a different output. But they definitely are cherry picking. So all this stuff you're seeing online is probably after hundreds and hundreds, if not thousands of generations, and them going, all right, here's the best ones we've come up with. All right, let's try again a little bit closer. No, it looks like a dog kind of running with a cape and it maybe flies, hovers for a second there, but definitely nothing close to what we're seeing in this image right here where it actually looks like a dog flying around with a cape. Let's see if we try a teddy bear running in New York City. Let's see if it looks anything like what their generation here looks like. All right, so here's our version of a teddy bear running through New York City. Not too bad, actually. That's actually one of the more impressive ones I've seen. Just keep in mind that a lot of the images that you're seeing from this text to video, you know, the ones that I showed you on Reddit here, that look really, really cool, the ones that they're showing off here on their Model Scope page here, these are the cherry picked ones. These are the ones where they probably tried hundreds of times, bunch of different seeds, until finally they came to a video that looked exactly like what they envisioned in their head. And so while this is really cool, you're probably going to need to do tons and tons and tons of prompts until you finally get one that looks like what you want it to look like. And every time you prompt one using your own T4 server like I am, it takes about one minute. So if you're trying to generate 20 different videos to finally get the video you're looking for, it might take you 20 minutes to generate 20 videos, but you likely can get to this level of quality that we see here and that we see here. It's just going to be a lot of trial and error until you do. But I want to remind you of something. This is super, super early tech. If we look back to when Dall? E 1 was first becoming available, back towards the end of 21, beginning of 2022, these are the types of images Dall? E 1 was generating less than a year ago. And here's what we can create today with things like Midjourney version 5. Here's some more early images from Dall? E, when text to image was just breaking through of an armchair that looks like an avocado. And here's the type of images we're generating today with something like Midjourney version 5. So in less than a year, we went from this to this. So if we're able to generate videos like this today, just imagine where this technology is going to be a year from now. Text to video is finally here. There's a version that you can play with. Yes, it may feel a bit underwhelming right now, and you might have to try a hundred times to get the exact video you're looking for. But it's here, it's available, and we're basically at day one of having access to this. Once again, I'll make sure the link to where this hugging face space is is in the comments below so you can go and use it. You probably will need to duplicate the space and upgrade your server in order to use it right now, but if you're one of the lucky ones that manages to get in and play around with it when the server isn't completely bogged down, you might be able to generate some images for free right now. Hopefully you enjoyed this quick video of an emerging tech that is brand new and as of right now, this is fresh. This is the hottest breaking thing right now in the AI world. If you're like most people and you feel like this AI space is just moving super, super fast and you want to just kind of stay in the loop a little bit, head on over to Future Tools IO. This is where I curate all of the coolest tools that I come across. I'm actually starting to remove some tools to make it less overwhelming. There's some junk tools on there, if I'm honest, and I'm starting to kind of get rid of some of them so that only the cream of the crop is being able to stay on the site. So check it out at FutureTools I and if what's on here is just still too overwhelming, still too many tools to kind of look through and you just want the TLDR of the week, Click here to join the free newsletter and every single Friday I'll just send you the five coolest tools that I came across and I'll give you the TLDR of the News and the coolest videos of the week as well as one cool way to make money with AI. I send it every Friday. All you gotta do is go to futuretools I oh. So thank you so much for tuning in. I'm gonna try to keep you up to date with all the latest and greatest in emerging technology in the AI space. So if you like this kind of video, give it a thumbs up. That'll make sure that you see more videos like this in your newsfeed and if you haven't already, subscribe to the channel that'll make sure you see more videos from me. I'm just so happy that there's so many other people that love nerding out over this AI tech like I do. And I just really appreciate you watching my videos to learn more about it. So thanks again for tuning in. Really appreciate you. See you in the next one. Bye.

Sam.