Dr. Károly Zsolnai-Fehér explains an AI that translates daytime photos into smooth time-lapse videos. The method trains on 20,000 unlabeled landscape images and optimizes seven objectives at once to generate multi-hour transitions with high detail.
April 22, 2020.
Approximately 20,000 landscape images.
Seven.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. A few years ago we have mainly seen neural network based techniques being used for image classification. This means that they were able to recognize objects, for instance animals and traffic signs in images. But today, with the incredible pace of machine learning research, we now have a selection of neural network based techniques for not only classifying images, but but also synthesizing them. The images that you see here and throughout this video is generated by one of these learning based methods. But of course, in this series we are always obsessed with artistic control, or in other words, how much of a say we have in the creation of these images. After all, getting thousands and thousands of images without any overarching theme or artistic control is, is hardly useful for anyone. One way of being able to control the outputs is to use a technique that is capable of image translation. What you see here is a work by the name Cyclegan. It could transform apples into oranges, zebras into horses, and more. It was called Cyclegan because it introduced a cycle consistency loss function. This means that if we convert a summer image to a winter image and then back to a summary image, we should get the same image back, or at least something very similar. If our learning system obeys this principle, the output quality of the translation is going to be significantly better. Today we are going to study a more advanced image translation technique that takes this further. This paper is amazingly good at daytime image translation. It looks at a selection of landscape images and then as you see here, it learns to reimagine our input photos as if they were taken at different times of the day. I love how clouds form and move over time in the synthesized images. And the night sky with the stars is also truly a sight to behold. But wait, Cyclegan and many other follow up works did image translation. This also does image translation. So what's really new here? Well, one, this work proposes a novel upsampling scheme that helps creating output images with lots and lots of detail. Two, it can also create not just a bunch of images a few hours apart, but it can also make beautiful time lapse videos where the transitions are smooth. Oh my goodness, I love this. And three, the training happens by shoveling 20,000 landscape images into the neural network and it becomes able to perform this translation task without labels. This means that we don't have to explicitly search for all the daytime images and tell the learner that these are daytime images and these other images are not. This is amazing because the algorithm is able to learn by itself without labels, but it is also easier to use because we can feed in lots and lots more training data without having to label these images correctly. As a result, we now know that this datetime translation task is used as a testbed to demonstrate that this method can be reused for other kinds of image translation tasks. The fact that it can learn on its own and still compete with other works in this area is truly incredible. Due to this kind of generality, it can also perform other related tasks. For instance, it can perform style transfer or in other words, not just change the time of day, but reimagine our pictures in the style of famous artists. I think with this paper we have a really capable technique on our hands that is getting closer and closer to the point where they can see use in mainstream software packages and image editors. That would be absolutely amazing. If you have a closer look at the paper, you will see that it tries to minimize seven things at the same time What a time to be alive. This episode has been supported by Weights and Biases. Here they show you how to build a proper convolutional neural network for image classification and how to visualize the performance of your model. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
