NVIDIA’s New AI: Paint Like Bob Ross!

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to have a look at Nvidia's new AI research work, which as they say, allows us to paint with words. So let's see. Yes, this runs a generative denoising process, or in other words, it starts out from a bunch of noise and over time uses our text prompt to rearrange it into an image that we described. Great. Then it subjects these coarse images to a super resolution technique, which means that in goes this coarse image and out comes an image with so much more detail and hence it can generate images of this quality. Now wait a minute. We are experienced fellow scholars over here, so we know that OpenAI's Dall E2 can do this and Google's Imagen can do this. The free and open source stable diffusion can also do this. So we have a number of papers that can pull this off really well. So is Nvidia a little late to the party? Why publish this paper? What is new here? Well, let's push it to its limits through three really fun experiments and find out together. 1. It gives us something that many of us fellow scholars desire and. And that is of course more control over the synthesized images. For instance, here we wish to create an image of boxing squirrels. Yes, you heard it right. But wait, here is the more granular control part. We can draw exactly where each squirrel and the boxing gloves go. And there we go, loving it. Or if we wish to create a rabbit who is also a magician, we. We can also specify that it should stand on clouds and we noted that it should cast a fireball. Now we can tell it where exactly that fireball should go. Two, it also follows our instructions really well when it comes to requesting styles as well. We can ask it to paint this penguin in the style of many famous artists with really cool results. However, sometimes just saying which artist we are looking for is not that helpful. You know, which phase of the artist are we talking about? Or which particular work should it be based on? And hold on to your papers because this can help even when our words fail us. How? Well, in this case we can also use an image instead. We can still add a text prompt and it will create a new image in the style of this one. This is especially useful in cases when we have a style in mind that is really hard to explain. I love this one. So cool. Three. So how does it compare to the usual suspects? Well, let's have a look at some teapots. Of course, to the surprise of no one stable Diffusion and dolly too are both capable of this task. However, look, we did not get a painting of a panda and with the new technique. Look at that. Oh yes, once again it follows our instructions better. Now note that text to image AIs are not easy to evaluate as all models can generate a ton of different images for the same prompt. However, further comparisons reveal the that there indeed is a pattern here. Now have you noticed there is a pattern in this video too? I keep saying that this new technique follows our instructions better. So here is the most important why how is all this wizardry possible? Well, this was one of my favorite parts of the paper. Have a look at this. The authors claim that as the classical text to image AIs start out from noise, the they follow our prompt closely. However, later on in the image synthesis process, not so much. If we change the prompt for the last few percentage of the noise diffusion process, look. Ouch. It completely ignores it. And here comes the best part. The authors trained multiple separate denoiser networks that are suited to different parts of the generation process. Hence yes, you guessed it right, these can follow our instructions better throughout later parts of the image generation process and thus give us better artistic control. Now I am sure that the next generation of text to image AIs are going to be even more powerful two more papers down the line, but this concept may live on to improve even the subsequent versions. I am very excited to see if this will really be the case. What a time to be alive. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance. And hold on to your papers. Because with Lambda GPU Cloud you can get on demand a 100 instances for $1.10 per hour versus $4.10 per hour with AWS. That's 73% savings. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Thanks for watching and for your generous support and see you next time.