Dr. Zsolnai explains how researchers edit AI image generators by rewriting model weights to change concepts across thousands of images at once. They swap towers for trees, add smiles, replace eyebrows with mustaches, and generate entirely new datasets from the edits.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Approximately seven months ago, we discussed an AI based technique called Stylegan 2, which could synthesize images of human faces for us. As a result, none of the faces that you see here are real. All of them were generated by this technique. The quality of the images and the amount of detail they're in is truly stunning. We could also exert artistic control over these outputs by combining aspects of multiple faces together. And as the quality of these images improve over time, we think more and more about new questions to ask about them. And one of those questions is, for instance, how original are the outputs of these neural networks? Can these really make something truly unique? And believe it or not, this paper gives us a fairly good answer to that. One of the key ideas in this work is that in order to change the outputs, we have to change the model itself. Now that sounds a little nebulous, so let's have a look at an example. First, we choose a rule that we wish to change. In our case, this will be the towers. We can ask the algorithm to show us matches to this concept, and indeed it highlights the towers on the images we haven't marked up yet, so it indeed understands what we meant. Then we highlight the tree as a goal, place it accordingly onto the water, And a few seconds later, There we go. The model has been reprogrammed such that instead of towers, it would make trees. Something original has emerged here. And look not only on one image, but on multiple images at the same time. Now have a look at these human faces. By the way, none of them are real and were all synthesized by Stylegan2, the method you saw at the start of the video. Some of them do not appear to be too happy about the research progress in machine learning. But I am sure that this paper can put a smile on their faces. Let's select the ones that aren't too happy, then copy a big smile and paste it onto their faces. See if it works. It does. Wow. Let's flick between the before and after images and see how well the changes are adapted to each of the target faces. Truly excellent work. And now on to eyebrows. Hold onto your papers while we choose a few of them. And now I hope you agree that this mustache would make a glorious replacement for them. And there we go. Perfect. And note that with this, we are violating Betteridge's law of headlines again in this series, because the answer to our central question is a resounding yes. These neural networks can indeed create truly original works. And what's more even entire datasets that haven't existed before now. At the start of the video we we noted that instead of editing images, it edits the neural network's model instead. If you look here, we have a set of input images created by a generator network. Then as we highlight concepts, for instance the watermark text here we can look for the weights that contain this information and rewrite the network to accommodate these user requests, in this case to remove these patterns now that they are gone. By selecting humans, we can again rewrite the network weights to add more of them, and finally the signature tree trick from earlier can take place. The key here is that if we change one image, then we have a new and original image. But if we change the generator model itself, we can make thousands of new images in one go, or even a full data set. Loving the idea. And perhaps the trickiest part of this work is minimizing the effect on other weights while we reprogram the ones we wish to change. Of course there will always be some collateral damage, but the results in most cases still seem to remain intact. Make sure to have a look at the paper to see how it's done exactly. Also good news. The authors also provided an online notebook where you can try this technique yourself. If you do, make sure to read the instructions carefully. And regardless of whether you get successes or failure cases, make sure to post them in the comment section here in Research. Both are useful information, so after the training step has taken place, neural networks can be rewired to make sure they create truly original works. And all this on not one image but on a mass scale. What a time to be alive. What you see here is an instrumentation of this exact paper we have talked about, which was made by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
