This Two Minute Papers YouTube episode (Two Minute Papers #68) explains how artistic style transfer—previously applied to still images—can be extended to videos by preserving temporal coherence. The speaker highlights that naively applying per-frame style transfer causes disturbing flicker because frames are processed independently; the proposed solution incorporates optical flow between frames so the stylization is propagated consistently across time, producing smooth, visually pleasing results. The video gives an intuition for temporal coherence (consistency across frames) and optical flow (per-pixel motion estimates between frames) and says the method 'preserves temporal coherence by incorporating the optical flow of the input video.'
Chapters
Dear fellow scholars, this is two minute papers with we have previously talked about a technique that used a deep neural network to transfer the artistic style of a painting to an arbitrary image, for instance to a photograph. As always, if you are not familiar with some of these terms, we have discussed them in previous episodes and links are available in the description box. Make sure to check them out. Style transfer is possible on still images. As there is currently no technique to apply this to videos, it is hopefully abundantly clear that a lot of potential still lies dormant inside. But can we apply this artistic style transfer to videos? Would it work if we would simply try? For an experienced researcher, it is flagrantly obvious that it's an understatement to say that it wouldn't work. It would fail in a spectacular manner, as you can see here. But with this technique, it apparently works quite well. To be frank, the results look gorgeous. So how does it work? Now, don't be afraid. You'll be presented with a concise but deliberately obscure statement. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video. Now the only question is what temporal coherence and optical flow means. Temporal coherence is a term that was used by physicists to describe, for instance, how the behavior of a wave of light changes or stays the same if we observe it at different times in computer graphics. It is also an important term because oftentimes we have techniques that we can apply to one image, but not necessarily to a video, because the behavior of the technique changes drastically from frame to frame, introducing a disturbing flickering effect that you can see in this video. Here we have the same if we do the artistic style transfer because there is no communication between the individual images of the video. The technique has no idea that most of the time we are looking at the same things, and if so, the artistic style would have to be applied the same way over and over to these regions. We are clearly lacking temporal coherence. Now onto optical flows. Imagine a flying drone that takes a series of photographs while hovering and looking around above us. To write sophisticated navigation algorithms, the drone would have to know which object is which across many of these photographs. If we have slightly turned, most of what we see is the same, and only a small part of this new image is new information. But the computer doesn't know that, as all it sees is a bunch of pixels. Optical flow algorithms help us achieving this by describing the possible motions that give us Photograph B from photograph A. In this application, what this means is that there is some inter frame communication. The algorithm will know that if I colored this person this way a moment ago and I cannot drastically change the style of that region on a whim, it is now easy to see why naively applying such techniques to many individual frames would be a flippant attempt to create beautiful smooth looking videos. So now it hopefully makes a bit more sense. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video. Such great progress in so little time. Loving it. Last time I've mentioned cram from the comment section and this time I'd like to commend relatedgerf for his insightful comments. Thanks for being around and I've definitely learned from you fellow scholars. I am really loving the respectful and quality discussions that take place in the comment section and it is really cool that we can both learn from each other. Thanks for watching and for your generous support and I'll see you next time.
