NVIDIA Cosmos - A Video AI…For Free!

This is an exclusive look at a huge new 76 page AI research paper on creating the future. And not just the future, but many thousands of futures, sort of. You see, this AI system has multiple models. One where we take an input image, add the text prompt and bam. It continues this image into the future as a video or these here are text to world results. No input image was used here, just a piece of text and a video of this quality comes out. And here comes the best part. Finally, this is not one of those fully closed systems. We all get this and can run it at home for free. And then you will see me run my own prompts before this technique was released. So these are results that you cannot see anywhere else. Just here. I would normally say that the visual quality is not as good as OpenAI Sora. However, this was made for a different purpose and it is really good at that. So you already see that these videos kind of have a theme. So what is all this about? Well, this work is about helping self driving cars and humanoid industrial or warehouse type robots learn about the world. The concept is brilliant. Here's why. You see, self driving cars have a long tail problem. This means that many normal scenarios have tons and tons of videos for the AI to learn how to behave. However, there are some crazy, crazy corner cases which have very few videos or not at all. For instance, traffic lights can safely be assumed to be stationary. They don't move of course, except when you see a truck transferring them, in which case they move. And thus the AI gets really really confused. It thinks what is going on here? This is one of those rare problems that are trivial to understand for us humans. But we need these crazy videos to teach an AI to understand them too. How can we deal with cases like that? Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Well, we can deal with this by being able to create thousands and thousands of continuations to help the AI get a detailed understanding of the situation. Or if you wish to teach a robot how to pick up an apple, you will need one video that starts with the robot picking up an apple. And you cannot teach this concept from just one video to a neural network. No sir, you need lots and lots of variants for that. And that is exactly what this technique is capable of. And I gotta say, if I am pixel peeping, I can see that these are not real videos. But there are cases which are really convincing for the first sight. So here are some incredible. This is not like OpenAI Sora. This is a model that is now available for all of us, for free, even for commercial use. The goal is that every AI has different kinds of hardware cameras, so they need different kinds of training data. And this system was done in a way that is easy to fine tune. They open sourced the code for that. So this enables us to create our own variants for our own use cases. And the whole research paper is available for free too. That describes how all this was done. Now, I had an early look at this paper and ran my own prompts to get these exclusive results to you fellow scholars. But we have two rooms rules. Rule 1 in these cases is that I always, always, always talk about the limitations of the paper to let you know about those two and two. No one can change what I say here. So the good news here is that the size of these models is not that huge, around 7 to 14 billion parameters, so they can run on a beefier laptop. However, the bad news is that generation times are not fast. A consumer graphics card has enough memory to run these models, but have to wait for about five minutes or longer for a few seconds worth of video footage. And when you are done, the quality of the results are not perfect. Not even close. It was meant to be a World foundation model for accurately simulating the future of a simulation, but there are cases where the apples don't fall correctly. Sometimes people can fly or grow six fingers on a hand, or both at the same time. And also object permanence is not guaranteed. Things may disappear over time. There is an autoregressive version of the technique that is much faster, but in return you have to make some more compromises in terms of visual quality. So there is plenty to improve for the next paper. I mean, if it is meant to be an accurate simulator of the future, it has to accurately simulate the future. However, I think this is a good time to invoke the first law of papers. The first law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. And based on historical data I would not be surprised if two or more papers down the line this becomes at least 10x if not a hundred x faster and more accurate at the same time. So cool. Please also note that we do not have a business relationship with Nvidia and this video was not sponsored by them. We have our own sponsors and this will likely be one of the puzzle pieces to make incredible new AI systems that will finally fold my laundry and finally not get lost in the same room after having it cleaned a hundred times before. All free to use and modify for all of us. What a time to be alive. The 76 page paper contains a ton of good including user study results to see what humans think about the quality of the results. Spoiler alert they favor it. Compared to previous techniques this paper was a herculean effort so I would like to send a huge thank you to all of the scientists who contributed to it to bring this to all of us for free. I put the links to it in the description. Let me know in the comments what you fellow scholars would use this for. To run your own experiments on an Nvidia gpu. Check out Lamda. I use it myself regularly for the these videos. Look at that. You can generate high quality images in less than a second per image. I did a ton more of them and paid less than a dollar for all this crazy. Seriously. Try it out now@lambdalabs.com papers or click the link in the description.