A new algorithm pastes parts of foreign images into paintings by applying style transfer and matching neural network responses across the whole image. In user studies, people preferred the result in 15 of 16 cases and couldn't tell when paintings had been edited.
May 15, 2018
15 out of 16 images
Yes — the presenter states the source code is available.
Dear fellow scholars, this is two minute papers with when we show a photograph to someone, most of the time we are interested in sharing our memories. Graduation, family festivities, beautiful landscapes are common examples of this. With the recent ascendancy of these amazing neural style transfer techniques, we can take a painting or any other source image and transfer the style of this image to our contents. The style is transferred but the contents remain unchanged. This takes place by running the images through a deep neural network network which in its deeper layers learns about high level concepts such as artistic style. This work has sparked a large body of follow up research feedforward, real time style transfer, temporally coherent style transfer for videos, you name it. However, these techniques are always about taking one image for content and one for style. How about a new problem formulation where we paste in a part of a foreign image with a completely different style? For instance, if you feel that this ancient artwork is sorely missing a Captain America shield, or if Picasso's self portrait is just not cool enough without shades, then this algorithm is for you. However, if we just drop in this part of a foreign image, anyone can immediately tell because of the differences in color and style. A previous non AI based technique does way better, but it is still apparent that the image has been tampered with. But as you can see here, this new technique is able to do it seamlessly. It works by first performing style transfer from the painting to the new region, and then in the second step additional refinements are made to it to make sure that the response of our neural network is similar across the entirety of the painting. It is conjectured that if the neural network is stimulated the same way by every part of the image, then there shouldn't be outlier regions that look vastly different. And as you can see here, it works remarkably well on a range of inputs. To validate this work, a user study was done that revealed that the users preferred the new technique over the older ones in 15 out of 16 images. I think it is fair to say that this work smokes the competition, but what about comparisons to real paintings? A different user study was also created to answer this question, and the answer is that users were mostly unable to identify whether the painting was tampered with. The source code is also available, so let the experiments begin. Thanks for watching and for your generous support and I'll see you next time.
