Dr. Zsolnai demonstrates Adobe's new AI video editing tool that separates foreground from background and treats 2D footage as 3D. You can stylize environments, remove objects, and change reflections by editing a single image, though it struggles with complex motion like wheel spokes and water wakes.
Dr. Zsolnai
Jan 13, 2022
Weights & Biases
WNB me paperintro (mentioned in the video description)
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to spice up this family video with a new stylized environment. And then we make a video of a boat trip much more memorable. Add the authenticity of a parkour video, decorate a swan, enhance our clothing a little, and conveniently forget about a speeding motorcycle. Only for scientific purposes, of course. So these are all very challenging tasks to perform. And of course none of these should be possible. And this is a new AI based solution that can pull off all of these. But how? Well, in goes an input video and this AI decomposes it into a foreground and background and. But in a way that it understands that this is just a 2D video that represents a 3D world. Clearly humans understand this, but does Adobe's new AI do it too? And I wonder how much it understands about that. Well, let's give it a try together. Put those flower on that dress. And what the flowers look like. They are really there as they wrinkle, as the dress wrinkles. And it catches shadows, just as the dress catches shadows too. That is absolutely incredible. It supports these high frequency movements so well that we can even stylize our kite sailing videos with it, where there is tons of tiny water droplets flying about, no problems at all. We can also draw on this dog. And remember we mentioned that it understands the difference between foreground and background. And look, the scribble correctly travels behind objects. Aha. And this is also the reason why we can easily remove a speeding motorbike from this security footage. Just cut out the foreground layer. Nothing to see here. But I wonder, can we go a little more extreme here? And it turns out these are really nothing compared to what this new AI can pull off. Look, we can not only decorate this swan, but here's the key. And oh yes, the swan is fine. Yes, but the reflection of the swan is also computed correctly. Now this feels like black magic, and we are not even done yet. Now hold on to your papers because here come my two favorite examples. Example number one, Biking in Wonderland. I love this. Now you see here that not even this technique is perfect. If you look behind the spokes of the wheel, you see a fair amount of warping. I still think this is a miracle that it can be pulled off at all. Example number two, a picturesque trip here. Not only the background has been changed, but even the water has changed as chunks of ice have also been added. And with that, I wonder how easy it is to do this. I'll tell you in a moment after we look at this. Yes, there is also one issue here with the warping of the water in the wake of the boat. But this is an excellent point for us to invoke the first law of papers which says do not look at where we are, look at where we will be. Two more papers down the line. So how much work do we have to put in to pull all this off? A day of work maybe? Nope, not even close. If you have been holding onto your paper so far. Now squeeze that paper and look at this. We can unwrap the background of the footage and stylize it as we please. All this takes is just editing one image. This goes in, this comes out. Wow. Absolutely anyone can do this. So this is a huge step in democratizing video stylization and bringing it to everyone. What a time to be alive. This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights and weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning and even hyperparameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link WNB me paperintro or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back. Thanks for watching and for your generous support and I'll see you next time.
