A short YouTube explainer (by LARRUZ) of the GANPaint paper and demo. The video describes a technique that dissects a generator neural network to find units aligned with visual concepts (trees, buildings, sky, clouds) and then edits images by activating/deactivating those units. It shows both global and region-constrained edits (adding/removing trees, bricks, doors), discusses limitations and failure cases (e.g., artifacts like a “spaghetti monster”), notes the work builds on GAN research, and highlights that the authors released source code and a web app for public use.
Chapters
Dear fellow scholars, this is two minute papers with this paper describes a new technique to visualize the inner workings of a generator neural network. This is a neural network that is able to create images for us. The key idea here is dissecting this neural network and looking for agreements between a set of neurons and concepts in the output image, such as trees, sky, clouds and more. This means analyzing that these neurons are responsible for buildings to appear in the image and those will generate clouds. Interestingly, such agreements can be found, which means way more than just creating visualizations like this, because it enables us to edit images without any artistic skills. And now hold on to your papers. The editing part works by forcefully activating and deactivating these units and correspond to adding or or removing these objects from an image. And look, this means that we can take an already existing image and ask this technique to remove trees from it or perhaps add more. The same with domes, doors and more. Wow, this is pretty cool, but you haven't seen the best part yet. Note that so far the amount of control we have over the image is quite limited. And fortunately we can take this further and select a region of the image where we wish to add something new. This is suddenly so much more granular and useful. The algorithm seems to understand that trees need to be rooted somewhere and not just appear from thin air most of the time anyway. Interestingly, it also understands that bricks don't really belong here, but if I add it to the side of the building, it continues in a way that is consistent with its appearance most of the time anyway. And of course it is not perfect. Here you can see me struggling with this spaghetti monster floating in the air that used to be a tree and it just refuses to be overwritten. And this is a very important lesson. Most research works are but a step in a thousand mile journey and each of them tries to improve upon the previous paper. This means that a few more papers down the line, this will probably take place in hd, perhaps in real time and with much higher quality. This work also builds on previous knowledge on generative adversarial networks. And whatever the follow up papers will contain, they will build on knowledge that was found in this work. Welcome to the wonderful world of research. And now we can all rejoice because the authors kindly made the source code available free for everyone. And not only that, but there is also a web app, so you can also try it yourself. This is an excellent way of maximizing the impact of your research work. Let the experts improve upon it by releasing the source code and let people play with it, even laymen. You will also find many failure cases, but also cases where it works well, and I think there is value in reporting both. So we learn a little more about this amazing algorithm. So let's do a little research together. Make sure to post your results in the comment section. I have a feeling that lots of high quality entertainment materials will surface very soon. I bet the authors will also be grateful for the feedback as well. So let the experiments begin. Thanks for watching and for your generous support and I'll see you next time.
