NVIDIA's NEW AI 'Text To Video Takes the Industry By STORM! (NOW UNVEILED!)

So Nvidia, the company that has been powering the AI race, has just released a new amazing tool. Check out this new research paper in which they document text to video. Now before I get into this insane research paper and all the examples and use cases, remember yesterday's video where there were many comments insinuating that text to video is far, far away and the rapid advancements won't be coming anytime soon. Well, just like clockwork, another new AI software research paper is is here and let's do a deep dive. So it starts out by saying high resolution video synthesis with latent diffusion models essentially just means that they're creating a text to video with models like stable diffusion. And you can see right here from the Sverta example, this is what we get. So the prompt is at the bottom, which I've added, and you can see right here that it says sunset time lapse at the beach, moving clouds and colors in the sky, 4K high resolution. I don't know about you guys, but I do think that this is actually pretty close to as realistic as you do get, which when it comes to text to video generation. Now there are some other examples that I am going to show you that aren't as good as this, but this just goes to show you how quickly the AI race is actually moving when it comes to things that we never thought were possible, like text to video. So let's talk a little bit more about how this stuff actually works and then we can get into some more of the examples because they've actually provided us with a ton of examples and there are a ton of things that are included in the research paper. You can see here from the abstract, there is a lot of text, but the piece that I've highlighted illustrates exactly what's going on. It says doing so we turn the publicly available state of the art text to image LDM stable diffusion into an efficient and expressive text to video model with a resolution of up to 1280 by 2048. So essentially what they're saying here is that they've turned stable diffusion into a text to video editor and the results seem to be pretty, pretty impressive. Now, I'm not sure when this is going to be released. I think they're still fine tweaking it, but take a look at some more of the other examples that they have provided us with. Before we do get to the end of the video, I want to know if you all notice a common theme about these certain types of videos, because I'm starting to realize that with text to video There's a certain kind of text to video that works, but there's a certain kind that doesn't. But anyways, let's take a look. So here's one of the first prompts. A teddy bear is playing on the guitar. High definition 4K. Now this was one of the ones that was one of their favorites. I guess it was a star favorite. And you can see that this is clearly an image that is based on stable diffusion, which is which they've used to generate this kind of video. Now of course this isn't the best kind of video, but it does show some sort of progress from what we did see in yesterday's video on Google's DreamX paper. And it's very interesting that they've actually used stable diffusion to do this because it seems like all these text to video editors are doing this in a different way. Now this is a interesting one because this is a stormtrooper vacuuming on the beach and I don't think this one is particularly bad. Now of course I'm going to state the obvious here by stating that this is by no means ready for film level production. But remember what Dall E2 was like in its early days and where it is now. We could certainly see some kind of improvements in this kind of video generation. Definitely a very interesting example. But take a look at this one. Now I got to be honest with you guys. This one right here, a fantasy landscape trending on ArtStation 4K. High resolution is by far one of the nicer ones. And I mean I, I do think that this is more on the side of, I wouldn't say realism, but I would say perhaps using it for maybe an in game loading screen, maybe some kind of cinematic, maybe a peaceful YouTube video. I'm not sure. It could definitely pass as one of those videos if maybe the quality was improved in certain areas. Now we also have another one here which is a turtle swimming in the ocean. And something that I've noticed about these videos released by these text to video models is, is that they seem to struggle with moving parts, I guess you could say moving animals. If there's something moving in the image, the image doesn't come out as good. But I guess you should be the judge of that because I think this one isn't as bad. But of course it isn't on midjourney's level quality yet. But nonetheless it still provides an early insight to what we might see in the future. Now this one I really do like because it is a close up of grapes on a Rotating table I High definition it actually does look pretty decent. I mean, one thing that I do realize that there is on a lot of these images is there is a little bit of grain which I would like to see removed in the upcoming versions. But nonetheless, I'm not going to lie, this one does actually seem pretty, pretty decent based on the text prompt that it was given. Now this one right here is definitely one of the more interesting ones that is honestly quite hilarious. I mean, it says a koala bear playing a piano in the forest. Honestly, it does kind of look like a koala bear playing a piano in the forest. And I mean, stay tuned for some other examples because there is something really cool that Nvidia did release with this software that I haven't shown you just yet. But I mean, this one is a hit and a miss, but it is definitely showing you how the AI is trying to work that out. Now this one right here is also definitely one of my favorites. It's a traveler walking alone in the misty forest at sunset. And you can see right here that you can. It does get the background very, very well. That misty sunset. It captures that type of scene perfectly, which is I guess, of course run by stable diffusion. But simply the motion of the character walking on the left isn't absolutely perfect. But like I said, with a few fine tunings, I'm pretty sure that eventually this is going to be perfect. Now here is the last one I'm going to show you before I talk about Nvidia's awesome release that they did with this as well. One is a time lapse of Aurora in the Sky 4K resolution. And like I said, it seems that when you do have these examples of fantasy landscapes or just simply time lapses, I think that this is where the video text to video kind of software does excel. Now there was something that they did include with this which I find to be very, very good because it does have some use cases. So let's take a look at what additionally they released with this research paper. So one thing that they did include was the idea of personalized video generation. So what I'm showing you on screen right now is dreamproof. And this is essentially fine tuning text to image to fusion models for subject driven video generation. So essentially this basically just means if you input a certain amount of images, you can actually then get certain images with that object in different places. So for example, in this example that we have on screen, you can see that the dog is then put into random location. You see that the dog is in a Doghouse. It's also in a bucket. It's also swimming underwater, it's also getting a haircut. And of course, this does provide a wide range of applications because, I mean, we all have our own objects and our own things that we would like to put into certain locations without having to go to those specific locations. So I can't imagine the amount of different things that you could literally do, do this with. Now, the reason I actually talked about Dream Booth, because of course it's something that already exists, is because they tried to do this with the AI video creator. So you can see right here that the training images that they used were that of Kermit the Frog. And then what we have here is, of course, the video. So they decided to use the text prompt, a SKS frog playing a guitar in a band. And if you're wondering what SKS stands for, it just means that specific object that's just the code word they use. So in this case, it would be Kermit the Frog playing a guitar in a band. And of course, you can see right here that although this doesn't look super realistic, we can see that the image generated output looks far more realistic than some of the moving ones that we saw initially in the beginning. So it's clear that when you provide that driving image, that your results are going to be far more superior. I mean, right here, Kermit the Frog writing a scientific research paper, this definitely does, once again, look more realistic. Now, I think what this does get clearly is that it does actually put the subject in the video clearly. And I think that's what is most important, because although the animations aren't completely perfect, and although the background isn't completely perfect, we do get a highly realistic visual of that specific character. Now, what's also interesting is that they did a very hard image here. You can see this teapot has multiple colors and many different ones, but somehow the AI actually does manage to get the, the text prompt correct. You can see right here an SKST pot floating in the ocean. And this one doesn't look that bad. I mean, the ripples do look pretty good. Although it is quite hard to stabilize that teapot. I think this output was very decent. Now, something that they also did was driving scene video generation, where they essentially trained the video LDM on real world driving video scene scenarios. And essentially what they were able to do is generate these videos. And some of these videos are literally up to five minutes, but here's the ones that they decided to provide. So here you can see this is a high resolution video of someone driving in I think America. Honestly, I'm not sure. I'm not like that guy that can just literally guess where you are in whatever country that you are. But I do think that these examples are very, very interesting. Now I do know that yes, there is some, I guess you could say distortion on the cars that do go by. But I think for dash cam videos it does seem pretty accurate. I mean, of course, like I said, distortion on the cars, there is a little bit of manipulation there on trees, the, and on the highways. But nonetheless, I still think that, you know, for example, this video right here is quite impressive in terms of what it's able to accomplish. What's also interesting is that they wanted to do specific driving scenario simulations. So essentially what they did is they, I guess they built up this image using these boxes and what they were able to do was recreate some kind of scenario where I guess they were able to create some kind of simulation for a specific scenario and, and the resulting video is what they came up with. So essentially what they're able to do is to create specific video scenarios based on what they wanted. Which is really cool because it goes to show that for the use case that, you know, maybe if you're a new driver and you want to just say, okay, what happens if I have three cars are about to crash into me? What do I do? How's that going to work? Maybe that could be a real use case. So definitely some very interesting stuff going on at Nvidia. Honestly, this stuff is moving very, very quickly. And one thing I do want to add is are we going to see a future where many of these text to video generators are going to be using mid journey instead of stable diffusion? Because we know that mid journey right now is pretty much at the stage where it doesn't really need any more improvement. I think it's completely done because it can pretty much do anything in terms of image generation, in terms of the quality. So it will be interesting to see where midjourney heads next and also where these other models head too.