Dr. Károly Zsolnai-Fehér walks through DALL-E 2, OpenAI's diffusion model trained on millions of image-caption pairs. It combines concepts to generate novel images, creates short videos through incremental prompts, renders large murals, and simulates photographic effects like subsurface scattering. The model could democratize art and spawn a new skill in prompt engineering.
And dear fellow scholars, this is 2 Minute Papers with Dr. Zsolnai. Today we are going to play some more with OpenAI's amazing technique Dall? E 2, where we can write a text description and the AI creates an amazing image of exactly that. That sounds cool. And it gets even cooler the crazier the ideas we give to it. This is an AI endowed with a diffusion based model, which means that when we ask it something, it starts out from noise. And over time it iteratively refines this image to match our description better. And over time, an absolutely incredible image emerges. This is a neural network that is given a ton of images and a piece of text description that says what is in this image? That is one image caption pair. Dall? E 2 is given millions and millions of these image pairs. So what can it do with all this knowledge? Well, the best part is that it can combine things. Oh yes, this is the key. This does not copy images from this training data, but it truly comes up with novel images. How? Well, after it had seen a bunch of images of koalas and separately a bunch of images of motorcycles, it starts to understand the concept of both and it will be able to combine the two together into a completely new image. And here you can see how much Dall? E 2 has improved since Dall? E 1. It is on a completely different level from its first iteration. This is so much better. And once again, this is an AI that was improved this much in just a year. I can hardly believe what I am seeing here. What a time to be alive. And now that some time has passed, new interesting capabilities have emerged. Now hold on to your papers and let's have a look at 10 more amazing examples. One, for instance, it can create not just images, but small videos. Videos how? Well, look at this video where a Victorian house is getting modernized. Wow. So what is going on here? Well, we can enter a text prompt, get an image and then change the text just a tiny bit, get a slightly changed image and repeat the process over and over until we get an amazing video like this. We can also run this process backwards and Victorianize a modern building as well. Two, if you don't believe that it can combine several existing concepts into something that definitely does not exist, hold on to your papers and have a Look at this. 1. Oh yes, Apple products, Leonardo da Vinci style. This is truly insane. If all this does not feel like human, like intelligence, I don't know what is. Three, here is how the AI imagines a robot learning a new language. Four, it can create new Kinds of drinks and my goodness. Well, I am super happy now. You are probably asking Karoj, why are you super happy? Well, I am a light transport researcher by trade and I spend a great deal of my time computing caustics. Oh yes, Caustics are these beautiful patterns that emerge when we hit a reflective or refractive object with light just the right way. And these can look especially magical if we have an object with a complex geometry. And the fact that the AI also understands this about our world. I am truly speechless. Beautiful. 5. It can also create new inventions. And this is a toilet car. You know, some people are quite busy and if you have to go when you are on the go, well, this one is for you. What a time to be alive. 6. In this one, the AI puts up a clinic in understanding combinations of concepts. Check this out. This is plants surrounding a light bulb. Now a light bulb surrounding some plants. Now a plant with a light bulb inside and a light bulb with plants inside. It can follow these instructions really well on all of these cases. 7. It can also create larger images too. So much so that entire murals can can be requested. And here not just the quality but the variety of the results is truly a sight to behold. 8. This is the famous Sith Lord Darth ant. Oh yes, this is Darth Vader reimagined as a robot ant. Loving it. 9. If you remember previously when we requested that it writes a piece of text on a sign, it floundered a great deal. This sign is supposed to say deep learning. And look, the amazing Peter Wellinger found a way to make it write things on signs properly. And all this with an amazing depth of field effect. Once again, this is an AI where a vast body of knowledge lies within, but it only emerges if we can bring it out with properly written prompts. It almost feels like a new kind of programming that is open to everyone, even people without any programming or technical knowledge. This is prompt engineering if you will. Perhaps a new kind of job that is just coming into existence. And 10 now check this out. We can even give instructions to it as a photographer would instruct its camera request mammatus clouds. We marveled together at the simulation of those in an earlier video and oh my goodness, that cannot be true. Look, the hand has subsurface scattering. What is that? That is the effect of light penetrating the skin, bouncing around and either coming out on the same or the other side. It has this absolutely beautiful halo effect. We worked on this a bit in an earlier paper together with Activision Blizzard and it took a great deal of mathematics and physics to perform this efficiently in a computer graphics engine. And now the AI just knows what it is. I really don't know what to say. And as always plus one because I couldn't resist. If you are interested in trying a reduced version of Dall? E, check this out. The link is available in the video description. Once again, please note that this is a highly reduced version of Dall? E, but it is still quite fun. Let the experiments begin. And I also cannot wait to get access to the full model. Some two minute papers, mascot figures and obviously images of white scholars holding onto their papers must come into existence. I am sure this tool will democratize art creation by putting it into the hands of all of us. We all have so many ideas and so little time and Dall? E will help with exactly that. So cool. Just imagine having an AI artist that is, let's say, just half as good as a human artist. But the AI can paint 10,000 images a day for free. Cover art for your album, illustrations for your novel. Not a problem. A little brain in a jar for everyone if you will. And now it's a really good one. Bravo. OpenAI. I am starting to believe that the impact of this AI is going to be so strong there will be the world as we know it before Dall? E and the world after it. So does this get your mind going? What else would you use this for? Let me know in the comments below.
This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insight. And weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning and even hyperparameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link WNB me paperintro or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back.
Thanks for watching and for your generous support and I'll see you next time.
