ControlNet adds precise control to Stable Diffusion by letting you feed it sketches, poses, or edge maps to guide image generation. It works on small datasets, runs free and open-source, and handles everything from video reenactment to style transfer without retraining the base model.
Dear fellow scholars, this is two minute papers with doctor doctor papers. Today we are going to look at an incredible way to supercharge stable diffusion, which is a free and open text to image. AI text goes in, beautiful images come out. We all know that. However, this is ControlNet, a new neural network structure that helps us give it additional inputs. What does that mean? More control. For instance, we can provide just the edges of an input image. This can be a rough sketch or edges extracted from a real photo. And bam. We get a beautiful image with exactly that creature and framing. Wow, that is exactly what I want. We can also provide this thing that we call a boundary map and ControlNet is able to follow that too. Do you know what else also works? Segmentation maps. This is just a rough draft of where the object should be. And my goodness, out comes a beautiful piece of landscape and architecture, or even an interior. And just look at that. All this came from these really coarse poor images. And don't forget, this works both ways. So we can even give this AI a photo, ask for the segmentation map, and then add a prompt to make it a painting, remove a tree or anything else. Pose works too. A stickman goes in and a photorealistic image comes out. That is incredible. So what else can we do with ControlNet? Well, after looking at these four amazing examples that I am about to show you, by the end of this video you will see that a more appropriate question would be, well, what can't we do with it? I will also tell you whether you can try it or not in a moment. Now hold on to your papers, fellow scholars, because this is going to be insane. One, with ControlNet, we can even transform ourselves. In goes a dancing video of us and out comes this. Is this really happening? I cannot believe this. Just look at how good the temporal coherence is. Yes, there are some unexpected changes from image to image. A little flickering, but this is improving so fast. Two more papers down the line. I bet my papers that no one is going to be able to tell. Just imagine how artists and even non artists will be able to unleash their creativity with this. Don't have an expensive camera to record this. Not a problem. No access to expensive lighting equipment. Not a problem. Heck, you don't even need to be at the location. It can be synthesized. That is incredible. Two, we can even create a group photo. And we can all enter a virtual world where we can be recreated with the same pose, character by character. How cool is that? So this was cartooning. And are you thinking what I am thinking, oh yes. 3. Decartooning is also possible. This way we can make cartoons and even our own drawn characters come to life. I really like how the skirt has been reimagined in a way that it could really exist in the real world. 4. Weights and Biases also asked the AI to reimagine their logo and it did not disappoint. Some of these are really imaginative. This is not just style transfer. Look at this animation example where the original footage is used as a control signal first. These indeed look like style transfer into watercolor or oil painting. Now look, it can truly reimagine the footage. Amazing. And the best part about ControlNet is whenever I am working with a tool like this, I. I feel more satisfied and more strongly involved in the creative process. I am not just prompting. I feel like I am at least partly creating something. I also feel like people are sleeping on this paper. They don't yet know that the world of digital art as we know it will change very quickly. More of us will have access to these fantastic tools that we could never afford to. And now it is available for all of us for free. The possibilities truly make my head spin. And this is just one paper. And this is a technique that is free and open to everyone. We build this together. So you fellow scholars are also welcome to contribute prompts, code, experiments, anything. This gives power to the people for free. That's what I want. Yes, I put a link to it in the video description and and to a guide too, so you can try it for free. And one more interesting tidbit from the paper. This neural network can even work with smaller datasets. So if we have less than 50,000 images, that is completely fine. Wow. But it can scale to billions of images if we so desire. And a big thanks to the Stable Diffusion subreddit as they host a ton of amazing artistic works with ControlNet. And now let the experiments begin.
This video has been supported by weights and biases.
Look at this.
They have a great community forum that aims to make you the best machine learning engineer you can be. You see, I always get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. In this forum you can share your projects, ask for advice, look for collaborators and more. Make sure to visit WNB Me Paperforum and say hi or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support. And I'll see you next time.
