Károly Zsolnai-Fehér explains how neural networks can generate large textures from tiny patches. A GAN trains by masking parts of images and asking a generator to fill them in, handling structured textures like wood rings and water. The results beat existing methods on both metrics and visual quality.
Dear fellow scholars, this is two minute papers with when an artist is in the process of creating digital media, such as populating a virtual world for an animation, movie or a video game, or even in graphic design, the artist often requires a large number of textures for these kinds of works. Concrete walls, leaves, fabrics are materials that we know well from the real world. And sometimes the process of obtaining textures is as simple as paying for a texture package and using it. But the problem quite often occurs that we wish to fill an entire road with a concrete texture, but we only have a small patch at our disposal. In this case, the easiest and worst solution is to copy paste this texture over and over, creating really unpleasant results that are quite repetitive and suffer from seams. So what about an AI based technique that looks at a small patch and automatically continues it in a way that looks natural and seamless? This is an area within computer graphics and AI that we call texture synthesis. Periodic texture synthesis is simple, but textures with structure are super difficult. The selling point of this particular work is that it is highly efficient at taking into consideration the content and symmetries of the image. For instance, it knows that it has to take into consideration the concentric nature of the wood rings when synthesizing this texture. And it can also adapt to the regularities of this water texture and create a beautiful high resolution. This is a neural network based technique. So first the question is what should the training data be? Let's take a database of high resolution images, let's cut out a small part and pretend that we don't have access to the bigger image and ask a neural network to try to expand this small cutout. This sounds a little silly. So what is this trickery good for? Well, this is super useful because after the neural network has expanded the results, we now have a reference result in our hands that we can compare to and and this way teach the network to do better. Note that this architecture is a generative adversarial network where two neural networks battle each other. The generator network is the creator that expands the small texture snippets. And the discriminator network takes a look and tries to tell it from the real deal. Over time, the generator network learns to be better at texture synthesis and the discriminator network becomes better at telling synthesized results from real ones. Over time, this rivalry leads to results that are of extremely high quality. And as you can see in this comparison, this new technique smokes the competition. The paper contains a ton of more results and comparisons and one of the most exhaustive evaluation sections I've seen in Texture synthesis so far, I highly recommend reading it. If you would like to see more episodes like this, make sure to pick up one of the cool perks we offer through Patreon, such as deciding the order of future episodes or getting your name in the video description of every episode. As a key supporter, we also support cryptocurrencies like Bitcoin, Ethereum, and Litecoin. We had a few really generous pledges in the last few weeks. I am quite stunned, to be honest, and I regret that I cannot come in contact with these fellow scholars. If you can contact me, that would be great. If not, thank you so much everyone for your unwavering support. This is just incredible. Thanks for watching and for your generous support and I'll see you next time.
