Dr. Zsolnai demos OpenAI's GLIDE, which generates images from text and can edit parts of existing images. It produces sharper, higher-resolution results than DALL·E and follows instructions better, according to a user study in the paper.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Today we are going to play with a magical AI where we just sit in our armchair, say the words and it draws an image for us. Almost anything we can imagine. Almost. And before you ask, yes, this includes drawing corgis too. In the last few years, OpenAI set out to train an AI named GPT3 that could finish your sentences. Then they made ImageGPT. This could even finish your images. Yes, not kidding. It could identify that the cat here likely holds a piece of paper and finish the picture accordingly. And even understood that if we have a droplet here and we see just a portion of the ripples, then this means a splash must be filled in. And it gets better. Then they invented an AI they call Dolly. This one is insanity. We just tell the AI what image we would like to see and it will draw it. Look, it can create a custom storefront for us. Understands the concept of low polygon rendering, isometric views, clay objects and more. And that's not all. It could even invent clocks with new shapes when asked. The crazy thing here is that it understands geometry, shapes and even materials. For instance, look at this white clock here on the blue table. And it not only put it on the table, but it also made sure to generate appropriate glossy reflections that matches the color of the clock. And get this. Dall E was published just about a year ago. And OpenAI already has a follow up paper that they call Glide. And believe it or not, this can do more and it can do it better. Well, I will believe it when I see it. So let's go. Now hold on to your papers and lets start with a hedgehog using a calculator. Wow, that looks incredible. It's not just a hedgehog plus a calculator. It really is using the calculator. Now paint a fox in the style of the Starry Night painting. I love the style and even the framing of the picture is quite good. There is even some space left to make sure that we see that Starry Night. Great decision making. Now a corgi with a red bow tie and a purple party hat. Excellent. And a pixel art corgi with a pizza. These are really good. But they are nothing compared to what is to come. Because it can also perform conditional inpainting with text. Yes, I am not kidding. Have a look at this. Little girl hugging a dog. But there is a problem with this. Do you know what the problem is? Of course. The problem is that this is not a corgi now it is that is another great result. And if we wish that some zebras were added here, that's possible too. And we can also add a vase here. Hmm, look at that. It even understood that this is a glass table and and added its own reflection. Now, I am a light transport researcher by trade, and this makes me very, very happy. However, it is also true that it seems to have changed the material properties of the table. It is now much more diffused than it was before. Perhaps this is the AI's understanding of a new object blocking reflections. It's not perfect by any means, but it is a solid step forward. We can also give this gentleman a white hat. And as I look through these results, I find it absolutely amazing how well the hat blends into the scene. That is very challenging. Why? Well, in light transport research, we need to simulate the path of millions and millions of light rays to make sure that indirect illumination appears in a scene. For instance, look here. This is one of our previous papers that showcases how fluids of different colors paint their diffused surroundings to their own color. I find it absolutely beautiful. Now let's switch the fluid to a different one. And yes, you see the difference. The link to this work is available in the video description below. And you see, simulating these effects is very costly and very difficult. But this is how proper light transport simulations need to be done. And this glide AI can put new objects into a scene and make them blend in so well. This, to me, also seems a proper understanding of light transport. I can hardly believe what is going on here. Bravo. But wait, how do we know if this is really better than Dolly? Are we supposed to just believe it? No, not at all. Fortunately, comparing the results against Dall E is very easy. Look, we just add the same prompts and see that there is no contest. The new glide technique creates sharper, higher resolution images with more detail. And it even follows our instructions better. The paper also showcases a user study where human evaluators also favored the new technique. Now, of course, we are not done here. Not even this technique is perfect. Look, we can request a cat with eight legs and wait a minute. It tried some multiplication trick, but we are not falling for it. A for effort, little AI. But of course, this is clearly one of the failure cases. And once again, this is a new AI where a vast body of knowledge lies within. But it only emerges if we can bring it out with properly written prompts. It almost feels like a new kind of programming that is open to everyone, even people without any programming or technical knowledge. If a computer is a bicycle for the mind. Then OpenAI's glide is a fighter jet. Absolutely incredible. Soon this might democratize creating paintings and maybe even help inventing new things. And here comes the best part. You can try it too. The notebook for it is available in the video Description make sure to leave your experiment results in the comments or just tweet them at me. I'd love to see what you ingenious fellow scholars bring out of this wonderful AI. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure. Plus, they are the only cloud service with 48GB RTX 8000. Join researchers at organizations like Apple, MIT, and Caltech in using Lambda Cloud instances with workstations or servers. Make sure to go to lambdalabs.com papers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their longstanding support and for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time.
