A short YouTube talk (Two Minute Papers) by LARRUZ reviewing recent machine-learning advances in image and video synthesis. It summarizes DeepMind's dual video discriminator approach for producing consistent two-second videos and introduces the concept of latent spaces as compressed representations that can be navigated to achieve artistic control. The video highlights new work that seeks nonlinear trajectories through latent space that correspond to intuitive edits (camera zoom, rotation, shifting, color enhancement, contrast), demonstrates partial success (e.g., zoom coupled with unwanted posture changes), and notes limitations from entangled latent movements and training-data constraints. The narrator points viewers to the paper linked in the description and to social media clips.
Chapters
Dear fellow scholars, this is two minute papers with as machine learning research advances over time, learning based techniques are getting better and better at generating images or even creating videos when given a topic a few episodes ago, we talked about DeepMind's dual video discriminator technique in which multiple neural networks compete against each other, teaching our machines to synthesize a collection of of two second long videos. One of the key advantages of this method was that it learned the concept of changes in the camera view zooming in on an object, and understood that if someone draws something with a pen, the ink has to remain on the paper unchanged. However, generally if we wish to ask an AI to synthesize assets for us, we'll likely have an exact idea of what we are looking for. In these cases, we are looking for a little more artistic control than this technique offers us. So can we get around this? If so, how? Well, we can. I'll tell you how in a moment. But to understand this solution, we first have to have a firm grasp on the concept of latent spaces. You can think of a latent space as a compressed representation that tries to capture the essence of the data set that we have at hand. You can see a similar latent space method in action here that set different kinds of fonts apart and presents these options on a 2D plane. And here you see our technique that builds a latent space for modeling a wide range of photorealistic material models that we can explore. And now onto this new work. What this tries to do is find a path in the latent space of these images that relates to intuitive concepts like camera zooming, rotation or shifting. That's not an easy task, but if we pull it off, we'll have more artistic control over these generated images, which will be immensely useful for many creative tasks. This new work can perform that, and not only that, but it is also able to learn the concept of color enhancement and can even increase or decrease the contrast of these images. The key idea of this paper is that this can be done through trying to find crazy natur nonlinear trajectories in these latent spaces that happen to relate to these intuitive concepts. It is not perfect in a sense that we can indeed zoom in on the picture of this dog, but the posture of the dog also changes and it even seems like we are starting out with a puppy that grows up frame by frame. This means that we have learned to navigate this latent space, but there is still some additional fat in these movements, which is a typical side effect of latent space based techniques. And Also, don't forget that the training data the AI is given also has its own limits. However, as you see, we are now one step closer to not only having an AI that synthesizes images for us, but one that does it exactly with the camera setup, rotation and colors that we are looking for. What a time to be alive. If you wish to see beautiful formulations of walks walks in latent spaces, I.e. make sure to have a look at the paper in the video description. Also note that we have now appeared on Instagram with bite sized pieces of our bite sized videos. Yes, it is quite peculiar. Make sure to check it out. Just search for 2 minute papers on Instagram or click the link in the video description. Thanks for watching and for your generous support and I'll see you next time.
