This YouTube explainer (Two Minute Papers, presenter Dr. Zsolnai) covers a new AI research paper that conditions video synthesis on a single reference video plus a text prompt, enabling users to provide motion/pose/framing by example rather than verbose textual directions. The video demonstrates transformations (e.g., astronaut surfing, surfing sloth, animal/environment swaps), highlights substantial qualitative and human-evaluation improvements over recent models (VDM, CogVideo) including better temporal coherence and reduced flicker, and notes a human evaluation on 32 videos favoring the new method. The clip includes a sponsor read for Weights & Biases.
Questions this video answers
2023-01-20
32 videos
Dr. Zsolnai (Two Minute Papers)
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to use this new AI research work to create absolutely magical videos almost out of thin air. Like this one and this one. Yes, with this new work, we can all become movie directors. Sort of loving it. Now, everyone knows that modern AI techniques are already capable of text to image. That is, taking a piece of text and creating a picture out of it. What fewer people know, but many of you fellow scholars definitely know, is that text to video is now also possible. However, I bet that you didn't know about today's paper. So what is this about? Well, whenever I use these systems, I always feel myself yearning for more artistic control. I want to specify the pose for these characters, the framing too. What should be visible, where the characters and objects are, and more. That would be lovely. So in this new paper, scientists had an excellent question. Why not give more information to the AI for these videos? And that is an excellent question. You see, this new technique allows us to show the AI exactly what we are looking for by performing it ourselves and giving it that as an input. It goes the following. Imagine that we would like to have a video of an astronaut surfing, but obviously that's not us. So we record ourselves surfing. Give this video to the AI, add a text prompt that now an astronaut should be surfing. And I am very excited. Let's see. Oh yes, that is so cool. And if we wish to reimagine what this scene would look like if we were surfing in the desert, not a problem. Or we can also think bigger and ask for a surfing sloth instead. Look at that. That sloth is having the time of his life. And I particularly like this example because this almost feels like a motion capture video where we just provide the movement for some other character. And all this with just one video and one text prompt. Wow. Now wait. This new work also gives us a ton more tools to play with. For instance, if we have a video of a bear playing a guitar, a common occurrence, we can also ask it to make it a little more artistic. And if we have a cat, we can make it too. Add cowboy hats, or even remove that hat and have it synthesize a similar video. I also loved this example. A simple video of this good boy running in the garden. And we can use it as the blueprint for our text to image ideas. Look, it has been changed to a Corgi and it is now in a different environment, or a lion or whatever we can imagine. So I hear you asking, what is so cool about this Example. Well, imagine if we are very picky with the kind of footage we want to synthesize. We would have to write in text what the pace direction and the exact angle should the walking take place in. But this paper says none of that is required. Just show the AI what you wish to see. So good. Now this is not the first paper to attempt this. So let's see what previous methods were made of. Oh boy. Look. This is the video diffusion model's paper. VDM in short. And the quality of the outputs is not the greatest. But there is an even bigger problem. What is the problem? Well, temporal coherence. Each adjacent frame seems so different. For instance, the color of the clothing is different each time. If we weave these together into a video, we will get a ton of flickering. Not good. So let's see the new method. Wow, this is so much better. Let's do another one of those fun comparisons. Let's see. Yes, this is cog video, another earlier technique. The prompt is a cat running on a single plank bridge. Comic style. Wow. I don't know about this one. Look, I am not sure if this is a cat, not sure if this is a bridge and I am not sure if this is in a comic style. The rest is fine though. So let's see the new one. Wow, this is incredible improvement in. Just wait. Actually, in how long exactly when were these two previous works published? Is this work 10 years of progress? No, no, not at all. And this is the part where I fell off the chair when reading this paper. Now hold on to your papers because both of these previous works were published less than a year before this new one. My goodness, that is amazing progress in just a few months. Months. Now we are experienced fellow scholars over here. So we are also looking for some more details regarding the old versus new technique. Let's dig a little deeper. And this is a human evaluation on a set of 32 videos. And are you seeing what I am seeing? This is one of the biggest improvements in just a few months that I have ever seen. Almost everyone favored the new one. Bravo. And just imagine what we will be able to do just a couple more papers down the line. Who knows, we all might become movie directors. What a time to be alive. So what do you think? What would you use this for? Let me know in the comments below.
Weights and Biases provides tools to track your experiments in your deep learning projects. What you see here is their amazing sweeps feature which helps you find and reproduce your best runs. And even better, what made this particular run the best. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in.
The video description and you can get.
A free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
