Károly Zsolnai-Fehér explains a neural network that transfers any art style to photos without training on specific styles first. The method compresses images through a bottleneck, then reconstructs them with style applied. It's fast enough for mobile and real-time use, with controls for how much style to apply.
Questions this video answers
Károly Zsolnai-Fehér
December 9, 2017
Less than a second
About one hour on a state-of-the-art graphics card (desktop GPU)
Chapters
Dear fellow scholars, this is two minute papers with. Let's have a look at some recent results on neural style transfer. You know the drill. We take a photo with some content and for example, a painting with the desired style, and the output is an image where this style is applied to our content. If this is done well and with good taste, it really looks like magic. However, for pretty much all the previous techniques, there are always some mysterious styles that result in failure cases. And the reason for this is the fact that these techniques are trained on a set of style images. And if they face a style that is wildly different from these training images, the results won't be very usable. This new algorithm is also based on neural networks and it doesn't need to be trained on these style images, but it can perform high quality style transfer and it works on arbitrary styles. This sounds a bit like black magic. So how does this happen exactly? First, an autoencoder is trained for image reconstruction. An autoencoder is a neural network where the input and output image is supposed to be the same thing. So far this doesn't make any sense because all the neural network does is copy and pasting the inputs to the outputs. Not very useful. However, if we reduce the number of neurons in one of the middle layers to very, very few neurons compared to the others, we get a bottleneck. This bottleneck essentially hamstrings the neural network and forces it to first come up with a highly compressed representation of an image. This is the encoder network. And then reconstruct the full image from this compressed representation. This is called the decoder network. So encoding is compression, decoding is decompression, or more intuitively, reconstruction. This compressed representation can be thought of as the essence of the image, which is a very concise representation which but carefully crafted such that a full reconstruction of the image can take place based on it. Autoencoders are previous work and if you would like to hear more about them, check the video description as we have dedicated an earlier episode to it. And now the value proposition of this work comes from the fact that we don't just use the autoencoder as is, but rip this network in half and use the encoder part on both the input style and content images. This way the concept of style transfer is much, much simpler in this compressed representation. In the end we are not stuck with this compressed result because if you remember, we also have a decoder which is the second part of the neural network that performs a reconstruction of an image from this compressed essence as a result, we don't have to train this neural network on the style images and it will work with any chosen style. Hell yeah. With most style transfer techniques we are given an output image and we either have to take it or leave it because we can't apply any meaningful edits to it. A cool corollary of this design decision is that we can also get closer to our artistic vision by fiddling with parameters. For instance, the scale and weight of the style transfer can be changed on the fly to our liking. As always, the new technique is compared to a bunch of other competing algorithms. Due to the general and lightweight nature of this method, it seems to perform more consistently across a set of widely varying input styles. We can also create some mattes for our target image and apply different artistic styles to different parts of it. Local parts of a style can also be transferred. Remember, the first style transfer technique was amazing but very limited and took an hour on a state of the art graphics card in a desktop computer. This one takes less than a second and works for any style. Now, as more new phones contain chips for performing deep learning, we can likely look forward to a totally amazing future where style transfer can be done in our pockets and in real time. What a time it is to be alive. Thanks for watching and for your generous support and I see you next time.
