This content was created by LARRUZ. This content was uploaded by LARRUZ. This is a YouTube video titled "NVIDIA Vid2Vid: AI-Based Video-to-Video Synthesis!" (2 Minute Papers), published 2018-09-09. It is a single-speaker English explainer in which the host reviews the Vid2Vid paper and demos how it improves on Pix2Pix for video synthesis. The host summarizes key contributions—using two discriminators (per-frame and video-level) to enforce temporal coherence, progressive training in space and time, and incorporation of flow maps to model frame-to-frame changes—shows examples (edge-to-face, label-to-scene, motion transfer), and notes results up to 2K/30s with source code available.
Chapters
Dear fellow scholars, this is 2 Minute Papers with Do you remember the amazing Pix2Pix algorithm from last year? It was able to perform image translation, which means that it could take a daytime image and translate it into a nighttime image, create maps from satellite images, or create photorealistic shoes from crew drawings. I remember that I almost fell off the chair when I first seen the results, but this new algorithm takes it up a notch and transforms these edge maps into human faces. Not only that, but it also animates them in time, as you see here. It also takes into consideration the fact that the same edges may result in many different faces, and therefore it is also willing to give us more of these options. If I fell out of the chair for the still image version, I don't really know what the appropriate reaction would be to this. It can also take a crude map of labels where each color corresponds to one object class, such as roads, cars or buildings, and it follows how our labels evolve in time and creates an animation out of it. We can also change the meaning of our labels easily. For instance, in the lower left you see how the buildings are now suddenly transformed to trees, or we can also change the trees to become buildings. Do you remember motion transfer from a couple of videos ago? It can do a similar variant of that too, and even synthesizes the shadows around the character in a reasonably correct manner. As you see, the temporal coherence of this technique is second to none, which means that it remembers what it did with past images and doesn't do anything drastically different for the next frame, and therefore generates smoother videos. This is very apparent especially when juxtaposed with the previous Pix2Pix method. So there are three key differences from the previous technique to achieve this. 1. The original architecture uses a generator network to create images where there is also a separate discriminator network that judges its work and teaches it to do better. Instead, this work uses two discriminator neural networks. One checks whether the images look good one by one, and one more discriminator for overlooking whether the sequence of these images would pass as a video. This discriminator cracks down on the generator network if it creates sequences that are not temporally coherent, and this is why we have minimal flickering in the output videos. Fantastic idea 2 to ease the training process, it also does it progressively, which means that the network is first faced with an easier version of the problem that progressively gets harder over time. If you have a look at the paper, you will see that the training is both progressive in terms of space and time. I love this idea too. 3 It also uses a flow map that describes the changes that took place since the previous frame. Note that this previous Pix2Pix algorithm was published in 2017 a little more than a year ago. I think that is a good taste of the pace of progress in machine learning research. Up to 2K resolution, 30 seconds of video and the source code is also available. Congratulations folks. This paper is something else. Thanks for watching and for your generous support and I'll see you next time.
