Dr. Károly Zsolnai-Fehér covers a new method that applies artistic styles to video in real time, preserving temporal consistency and detail better than previous approaches. The technique supports partial keyframe editing but still struggles with temporal flicker and foreground-background separation.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Style transfer is an interesting problem in machine learning research where we have two input images, one for content and one for style and the output is our content image reimagined with this new style. The cool part is that the content can be a photo straight from our camera and the style can be a painting which leads to super fun and really good looking results. We have seen plenty of papers doing variations of style transfer, but I always wonder can we push this concept further? And the answer is yes. For instance, few people know that style transfer can also be done for video. First we record a video with our camera, then take a still image from the video and apply our artistic style to it. Then our style will be applied to the entirety of the video. The main advantage of this new method compared to previous ones is that they either take too long or we have to run an expensive pre training step. With this new one we can just start drawing and see the output results right away. But it gets even better due to the interactive nature of this new technique, we can even do this live. All we need to do is change our input drawing and it transfers the new style to the video and as fast as we can draw. This way we can refine our input style for as long as we wish or until we find the perfect way to stylize the video. And there is even more. If this works interactively, then it has to be able to offer an amazing workflow where we can capture a video of ourselves live and mark it up as we go. Let's see. Oh wow, just look at that. It is great to see that this new method also retains temporal consistency over a long time frame. Which means that even if the marked up keyframe is from a long time ago, it can still be applied to the video and the outputs will show minimal flickering. And note that we can not only play with the colors that but with the geometry too. Look, we can warp the style image and it will be reflected in the output as well. I bet there is going to be a follow up paper on more elaborate shape modifications as well. And this new work improves upon previous methods in even more areas. For instance, this is a method from just one year ago and here you see how it struggled with contour based styles. Here's a keyframe of the input video and here's the style that we wish to apply to it later. This method from last year seems to lose not only the contours but a lot of visual detail is also gone. So how did the new method do in this case look, it not only retains the contours better, but a lot more of the sharp details remain in the outputs. Amazing. Now note that this technique also comes with some limitations. For instance, there is still some temporal flickering in the outputs and in some cases separating the foreground and the background is challenging, but really such incredible progress in just one year and I can only imagine what this method will be capable of two more papers down the line. What a time to be alive. Make sure to have a look at the paper in the video description and you will see many additional details. For instance how you can just partially fill in some of the keyframes with your style and still get an excellent result. This episode has been supported by Weights and Biases. In this post they show you how to test and explore putting bounding boxes around objects in your photos. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
