Google’s New AI: OpenAI’s DALL-E 2, But 10X Faster!

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see progress in text to image research that is so incredible it hardly seems believable. What is text to image? Simple. These are AI based techniques where a text prompt from us goes in and a beautiful image comes out. There are already a large set of techniques that can perform text to image really well. For instance, OpenAI's Dall E2 can do it where we wait approximately 10 seconds for each image. Or we can even run it on our own hardware with the free and open source stable diffusion. So are we done? What else is there to invent here? Why write more papers? Well, by the time you finish this video, I hope you will agree with me that the only possible answer is my goodness, there is so much to be done and so much that has just been improved. For instance, this new technique from Google that they call Muse can perform mask free editing too. What is that? To be able to appreciate what that is, let's look at what mask based editing looks like. Imagine that we have a wonderful photo here, but we would like to change the location of the background. No matter, let's just highlight this region which we will call a mask and ask for a different background. And yes, it can repaint the image as if it were in New York, Paris or San Francisco. Wonderful. This is image inpainting by using a mask and a text prompt. And now let's do this. But without a mask. What? How is this even possible? Well, we can tell the. Come on, you are smart enough to know where these objects are and what they are. So you do the masking yourself automatically. So can it. Let's see. There is a cake in this image and a cafe latte. But.

Psst.

Don't tell the AI confidential information. Let it find out itself. Now little AI, please change the cake to a croissant. If it even exists, who knows. And the latte art should form a flower and. Wow, look at that. Who could say that this was not the original image it has done absolutely amazing. And I am a light transport researcher by trade, so I cannot resist mentioning that the specular highlights on the new plate are also excellent. Good job, Little AI. But its mask free editing capabilities can do even better. Look, we can change our clothes super easily, even with text. Remember, synthesizing text properly was quite difficult for previous techniques. And this just does it easily. So good. Now get this. Because we can also do this with drawings. And if we do something amazing happens. Look, we can start out from a crude drawing of a cat and ask it to morph into different animals. And through this it can become a dog or a pig, a raccoon or even other animals. This is possible because this new technique is not your usual diffusion based process like many previous image generators. What does that mean? It means that it does not start out from noise and does not reorganize this noise to get a coherent image. It does not think in terms of noise at all. But who we are. Not done yet. Not even close. It can also perform image outpainting. That is taking this part of the image and replacing the entirety of the image around it using a text prompt. Travel around the world with just one text prompt. So cool. Now have a look at these images and their prompts. Great works, right? But there is something that ties them together. Do you know what? Well, hold on to your papers. Because all of these Images took approximately 1 second to generate. That's right. This is up to 10 times faster than previous techniques. And all this less than a year after Dall? E 2 has been published. This is insanity. And it can also perform things that that other previous techniques had a great deal of trouble with. What are those? Well, two. Cardinality and composition. Cardinality means that if we ask for three elephants standing on top of each other, we really get three. And if we ask for four bottles of wine, we get four. And if we ask for 10 bottles of wine. Wait a minute. Yes. Apparently not even this technique is perfect. It also does well when it comes to composition. If we ask for the two baseballs to be to the left of the tennis balls, the AI understands that, and thus they will likely end up being there. However, I also love how it combines all of these concepts together. For instance, here we can do mask free editing while keeping the composition of the original image the same. This way we can transform our cat into a dog, Change a small basketball into an American football. Or make our cat yawn. Or even change these flowers. And note that we did not need a mask for this. No highlighting regions where the cats and the roses are. We just write what we want. And the AI does it also note once again that the composition of the original image remains intact? I absolutely love this. Such an amazing tool. And now a really fast one too. One second for each of these. Sign me up right now. And just imagine that two more papers down the line. Perhaps all this will be possible to do in real time. We might be able to create little virtual worlds with the speed of thought. How cool is that? What a time to be alive. So what do you think? What would you use this for? Let me know in the comments below.

This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights. And weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning and even hyper parameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link wnb me or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back.

Thanks for watching and for your generous support and I'll see you next time.