A short YouTube explainer by Dr. Zsolnai (Two Minute Papers) describing a new NVIDIA-related technique that lets users "paint with images" by using diffusion-based generative models as brushes. The video explains the idea (feeding an image as a brush instead of pixel brushes), demonstrates non-repetitive tiling and controlled placement, compares the method to prior approaches (listing three claimed advantages), and notes potential workflows combining this with text-to-image models and accessible open tools like Stable Diffusion.
Chapters
Today we are going to use an AI to paint. Now wait a minute, that doesn't make any sense, does it? Especially that on your tablet you can use these painting apps where you choose a brush and use your pencil and pretend that it is a real paintbrush. These apps are available right now, so why use an AI? Because today we are going to put a twist on it. Yes, we are going to paint not with brushes, but with images. Oh yes. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. OK, so what is this insanity? Well, imagine a diffusion based AI technique for generated images from text. We start out from noise and over time diffusion means it reorganizes this noise and into an image. So here is an insane idea. Let's not use brushes anymore like before, but plug in an image instead and use our pencil with that. That sounds fantastic. But there is a problem. You see if we have an image and we just tile the space with copies of that image. Uh oh. This is really boring and repetitive. That's not what we are looking for. And now let's see what painting with images is like with this new technique. This is a nice little hut, but a rocky road leading to it would be even nicer. And we grab that brush and oh, we got something. Let's see. It comes from an image and it is not repeating. Great. We can also use it to paint mass on an object as well. And we can really control it to make sure it has the right amount and at the right places. Now wait, wait, wait. This is not the first technique to do such a thing. Not at all. So how does it compare to its predecessors? Well, it is better in three ways. One, against Photoshop's technique, it tiles way better, especially on something that it hasn't seen yet. Two, consistency is particularly difficult, especially if we draw lines that have a bit of a drift. The AI has to be really mindful to make it look believable. Previous techniques had issues making this really consistent, but this one looks so cool. And three, unlike previous systems, this technique now allows us to draw with almost any image. Wow. So now with this we can also tie the roof with all this sweetness or if we wish to create a tree and we are looking to have a little more control over how each part looks. This is what we get and we don't need to be a trained specialist to do that. So cool. And while we create this little toy dinosaur, I will note that clearly the results are not as good as we would expect from a real human artist. Not nearly as good. An expert artist can do way better. However, the potential is huge. For instance, what I think a future paper will do is take the entirety of the image now and give it to a powerful text to image AI to generate a new one with this as a guideline like this workflow here. So when the dress is done, you take what you have drawn and get a better one. And you know what the coolest part is? You may be able to do something like that right now by using Stable Diffusion and other free and open tools. Absolutely amazing. And just think about what we will be able to do two more papers down the line. That is the first law of papers and I love how this enables the rest of us to be a little more creative and make something really cool today. Loving it. The paper also contains more detailed comparisons against more previous techniques, so make sure to have a look in the description. It is super fun. Also, according to the website, the code is supposedly also available or will be available. Here's hoping. And what do you think? What would you fellow scholars use this for? Let me know in the comments below. If you're looking for an Nvidia GPU cloud service for large scale AI projects, check out One Click Clusters by Lamda, featuring up to 512 Nvidia H100 GPU GPUs with InfiniBand networking. And hold on to your papers because with one click clusters you can access all these GPUs on demand with reservations starting at just two weeks. Try it out@lambdalabs.com paper or click the link in the description.
