A Two Minute Papers video (Jan 17, 2017) where Sam summarizes a research paper that trains generative adversarial networks on large collections of unlabeled Internet videos to synthesize short videos. The work uses a two-stream generator that models foreground and background separately and a discriminator to judge realism; user studies indicate preference for this method over prior approaches. Outputs are low-resolution (64×64 pixels, 32 frames) and vary in quality, but the network shows emergent understanding of object movement; neuron-activation visualizations and an authors' website with examples are mentioned.
Questions this video answers
64 × 64 pixels for 32 frames.
Chapters
Dear fellow scholars, this is two minute papers with ever thought about the fact that we have a stupendously large amount of unlabeled videos on the Internet. And of course with the ascendancy of machine learning algorithms that can learn by themselves, it would be a huge missed opportunity to not make use of all this free data. This is a crazy piece of work where the idea is to unleash a neural network on a large number of publicly uploaded videos on the Internet which and see how well it does if we ask it to generate new videos from scratch. Here, unlabeled means there is no information as to what we see in these videos. They are just provided as is machine learning methods that work on this kind of data we like to call unsupervised learning techniques. This work is based on a generative adversarial network. Wait, what does this mean exactly? This means that we have two neural networks that race each other where one tries to generate more and more real looking animations and passes it over to the other that learns to tell real footage from fake ones. The first we call the generator network and the second is the discriminator network. They try to outperform each other and this rivalry goes on for quite a while and improves the quality of output for both neural networks. Hence the name generative adversarial networks. At first we have covered this concept that was used to generate images from written text descriptions. The shortcoming of this approach was the slow training time that led to extremely tiny low resolution output images. This was remedied by a follow up work which proposed a two stage version of this architecture. We have covered this in an earlier two minute Papers episode. As always, the link is available in the video description. It would not be an understatement to say that I nearly fell off the chair when seeing these incredible results. So, where do we go from here? What shall be the next step? Well, of course video. However, the implementation of such a technique is far from trivial. In this piece of work, the generator network learns not on the original representation of the videos, but on the foreground and background video streams separately and it also has to learn what combination of these yields realistic footage. This two stream architecture is particularly useful in modeling real world videos where the background is mostly stationary and and there is an animated movement in the foreground. A train passing the station or people playing golf on the field are excellent examples of this kind of separation. We definitely need a high quality discriminator network as well as in the final synthesized footage, not only the foreground and background must go well together, but the synthesized animations also have to be believable for human beings. This human being in our case is represented by the discriminator network. Needless to say, this problem is extremely difficult and the quality of the discriminator network makes or breaks this magic trick. And of course the all important question immediately if there are multiple algorithms performing this action, how do we decide which one is the best? Generally we get a few people and show them a piece of synthesized footage with this algorithm and previous works and have them decide which they deem more realistic. This is still the first step. I expect these techniques to improve so rapidly that we'll soon find ourselves testing against real world footage and who knows, sometimes perhaps failing to recognize which is which. The results in the paper show that this new technique beats the previous techniques by a significant margin and that users have a strong preference towards the two stream architecture. The previous technique they compare against is an autoencoder which we have discussed in a previous two minute Papers episode. Check it out, it is available in the video Description the disadvantages of this approach are quite easy to identify. This time around we have a very limited resolution for these output video streams I.e. 64 x 64 pixels for 32 frames, which even at a modest frame rate is just slightly over one second of footage. The synthesized results vary greatly in quality, but it's remarkable to see that the machine can have a rough understanding of the concept of a large variety of movement and animation types. It is really incredible to see that the neural network learns about the representations of these objects and how they move, even when it wasn't explicitly instructed to do so. We can also visualize what the neural network has learned. This is done by finding different image inputs that make a particular neuron extremely excited. Here we see a collection of inputs including these activations for images of people and trains. The author's website is definitely worthy of checking out as some of the sub menus are quite ample in results. Some amazing, some, well, a bit horrifying. But what is sure is that all of them are quite interesting. And before we go, a huge shout out to Laszlo Cendas, who helped us quite a bit in sorting out a number of technical issues with the series. Thanks for watching and for your generous support and I'll see you next time. Sam.
