Dr. Zsolnai walks through OpenAI's DALL-E, a text-to-image model that can generate storefronts with brand text, invent objects, and render in specific artistic styles. It evolved from GPT-2 and GPT-3, and despite current limitations, the speaker sees major potential in future access to pretrained models.
Dear fellow scholars, this is 2 Minute Papers with Dr. Zsolnai. In early 2019, a learning based technique appeared that could perform common natural language processing operations. For instance, answering questions, completing text, reading comprehension, summarization, and more. This method was developed by scientists at OpenAI and they called it GPT2. The key idea for GPT2 was that all of these problems could be formulated as different variants of text completion problems, where all we need to do is provide it an incomplete piece of text and it would try to finish it. Then in June 2020 came GPT3 that supercharged this idea. And among many incredible examples, it could generate website layouts from a written description. However, no one said that these neural networks can only deal with text information. And sure enough, a few months later, scientists at OpenAI thought that if we can complete text sentences, why not try to complete images too? They called this project ImageGPT. And the problem statement was simple. We give it an incomplete image and we ask the AI to fill in the missing pixels. It could identify that the cat here likely holds a piece of paper and finish the picture accordingly. And even understood that if we have a droplet here and we see just a portion of the ripples, then this means a splash must be filled in. And now, right in January 2021, just seven months after the release of GPT3, here is their new mind blowing technique that explores the connection between text and images. But finishing images already kind of works. So what new thing can it do? In just a few moments you will see that the more appropriate question would be what can't it do? For now, well, it creates images from our written text captions and you will see in a moment how monumental of a challenge that is. The name of this technique is a mix of Salvador Dali and Pixar's Wall E. So please meet Dolly. And now let's see it through an example for neural network based learning methods. It is easy to recognize that this text says OpenAI and what a storefront is. Images of both of these exist in abundance. Understanding that is simple. However, generating a storefront that says OpenAI is quite a challenge. Is it really possible that it can do that? Well, let's try it. Look, it works. Wow. Now, of course, if you look here, you immediately see that it is by no means perfect. But let's marvel at the fact that we can get all kinds of 2D and 3D texts, look at the storefronts from different orientations, and it can deal with all of these cases reasonably well. And of course, it is not limited to storefronts. We can request license plates, bags of chips, neon signs and more. It can really do all that. So what else? Well, get this. It can also kind of invent new things. So let's put our entrepreneurial hat on and try to invent something here. For instance, let's try to create a triangular clock or pentagonal, or you know, just make it a hexagon. It really doesn't matter because we can ask for absolutely anything and get a bunch of prototypes in a matter of seconds. Now let's make it white. And look, now we have a happy, happy caroi. Why is that? It is because I am a light transport researcher by trade, so the first thing I look at when seeing these generated images is how physically plausible they are. For instance, look at this white clock here on the blue table. And it not only put it on the table, but it also made sure to generate appropriate glossy reflections that matches the color of the clock. It can do this too. Loving it. Apparently it understands geometry, shapes and materials. I wonder what else does it understand? Well, get this for instance. It even understands styles and rendering techniques. Being a graphics person, I am so happy to see that it learned the concept of low polygon count rendering, isometric views, clay objects, and we can even add an X ray view to the owl, kind of. And now if all that wasn't enough, hold on to your papers because we can also commission artistic illustrations for free. And not only that, but even have fine grained control over these artistic illustrations. I also learned that if manatees wore suits, they would wear them like this. And and after a long and strenuous day walking their dogs, they can go for yet another round in pajamas. But it does not stop there. It can not only generate paintings of nearly anything, but we can even choose the artistic style and the time of day as well. The night images are a little on the nose as most of them have the moon in the background. But I'll be more than happy to take these. And the best part is that you can try it yourself right now through the link in the video description. In general, not all results are perfect, but it's hard to even fathom all the things this will enable us to do in the near future when we can get our hands on these pre trained models. And this may be the first technique where the results are not limited by the algorithm, but by our own imagination. Now this is a quote that I said about GPT3 and notice that the exact same thing can be said about Dolly quote. The main point is that working with GPT3 is a really peculiar process where we know that a vast body of knowledge lies within, but it only emerges if we can bring it out with properly written prompts. It almost feels like a new kind of programming that is open to everyone, even people without any programming or technical knowledge. If a computer is a bicycle for the mind, then GPT3 is a fighter jet. Absolutely incredible. And I think this kind of programming is going to be more and more common in the future. Now note that these are some amazing preliminary results, but the full paper is not available yet. So this was not two minutes and it was not about a paper. Welcome to Two Minute Papers. Jokes aside, I cannot wait for the paper to appear and I'll be here to have a closer look whenever it happens. Make sure to subscribe and hit the bell icon to not miss it when the big day comes. Until then, let me know in the comments what crazy concoctions you came up with. Perceptilabs is a Visual API for TensorFlow, carefully designed to make machine learning as intuitive as possible. This gives you a faster way to build out models with more transparency into how your model is architected, how it performs, and how to debug it. Look, it lets you toggle between the visual modeler and the code editor. It even generates visualizations for all the model variables and gives you recommendations both during modeling and training, and does all this automatically. I only wish I had a tool like this when I was working on my neural networks during my PhD years. Visit perceptilabs.com papers to easily install the free local version of their system today. Our thanks to Perceptilabs for their support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
