Short video (Two Minute Papers #80) that summarizes a paper which maps crude hand-drawn sketches to photorealistic images by retrieving similar photos from a database rather than synthesizing new images. The learning approach trains deep convolutional neural networks on paired photographs and sketches (mentions Siamese and triplet network techniques) and relies on a newly released paired dataset provided free by the authors. The speaker points viewers to an online demo and a subreddit for sharing results.
Chapters
Dear fellow scholars, this is two minute papers with when we were children, every single one of us dreamed about having a magic pencil that would make our adorable little drawings come true. With the power of machine learning, the authors of this paper just made our dreams come true. Here's the we provide a crude drawing of something and the algorithm fetches a photograph from a database that depicts something something similar to it. It's not synthesizing new images from scratch from a written description like one of the previous works. It fetches an already existing image from a database. The learning happens by showing a deep convolutional neural network, pairs of photographs and sketches. If you are not familiar with these networks, we have some links for you in the video description box. It is also important to note that this piece of work does not showcase a new learning technique. It is using existing techniques on a newly created datab that the authors kindly provided free of charge to encourage future research in this area. What we need to teach these networks is the relation of a photograph and a sketch. For instance, in an earlier work by the name Siamese Networks, the photo and the sketch would be fed to two convolutional neural networks with the additional information whether this pair is considered similar or dissimilar. This idea of Siamese networks was initially applied to signature verification more than 20 years ago. Later, triplet networks were used to provide the relation of multiple pairs. Like this sketch is closer to this photo than this other one. There is one more technique referred to in the paper that they used, which is a quite delightful read. Make sure to have a look. We need lots and lots of these pairs so the learning algorithm can learn what it means that a sketch is similar to a photo and as a result fetch meaningful images for us. So if we train these networks on this new database, this magic pencil dream of ours can come true. What's even better, anyone can try it online. This is going to be a very rigorous and scholarly scientific experiment. I don't know what this should be, but I hope the algorithm does well. That kinda makes sense. Thanks Algorithm. For those fellow scholars out there who are endowed with better drawing skills than I am, or, well, basically all of you, if you have tried it and got some amazing or maybe not so amazing results, please post them in the comment section. Or, as we now have our very own subreddit, make sure to drop by and post some of your results there so we can marvel at them or have a good laugh at possible failure cases. I am looking forward to meeting you fellow scholars at the subreddit flares are also available. Thanks for watching and for your generous support, and I'll see you next time.
