NVIDIA’s New AI: Superb Details, Super Fast! 🤖

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to look at Nvidia's spectacular new AI that can generate beautiful images for us. But this is not image generation of any kind. No, no, this is different. Let's see how it is different. For instance, we can write a description and the appropriate image comes out. Snowy mountains, pink cloudy sky. Check mark. Okay, so we can give it our direction. Make no mistake, this is fantastic. But this has been done before, so nothing new here yet. Or we can create a segmentation map. This tells the AI what things are. The sea is down there, mountains and sky up here looking great. But this has been done before too. For instance, Nvidia's previous Gauguin paper could do this too. Nothing new here yet. Or we can tell the AI where things are by sketching. This one also works, but this has been done too. So is there nothing new in this paper? Well, of course there is. And now hold on to your papers and watch as we fuse all of these descriptions together. We tell it where things are and what things are. But I wonder if we can make this mountain snowy and a pink cloudy sky on top of all things. Yes, we can. Oh wow. I love it. The sea could have a little more detail, but the rest of the image is spectacular. So with this new technique, we we can tell the AI where things are, what things are. And on top of it, we can also give it art direction. And here's the key. All of these can be done in any combination. Now, did you see it? Curiously, an image popped up at the start of the video when we unchecked all the boxes. Why is that? Is that a bug? I'll tell you in a bit what that is. So far, these were great examples. But let's try to push this to its limits and see what it can do. For instance, how quickly can we iterate with this? How quick is it to correct mistakes or improve the work? Oh boy, super quick. When giving art direction to the AI, we can update the text and the output refreshes almost as quickly as we can type. The sketch to. Image feature is also a great tool by itself, of course, there is not only one way. There are many pictures that this could describe. So how do we control what we get? Well, it can even generate variants for us. With this new work, we can even draw a piece of rock within the sea, and the rock will indeed appear. And not only that, but it understands that the waves have to go around it too. An understanding of physics. That is insanity. My Goodness. Or better. If we know in advance that we are looking for tall trees and autumn leaves, we can even start with the art direction and then when we add our labels, they will be satisfied. We can have a river, okay, but the trees and the leaves will always be there. Finally, we can sketch on top of this to have additional control over the hills and clouds. And get this, we can even edit real images. So how does this black magic work? Well, we have four neural networks, four experts if you will. And the new technique describes how to fuse their expertise together into one amazing package. And the result is a technique that outperforms previous techniques in most of the tested cases. So are these some ancient methods from many years ago that are outperformed or are they cutting edge? And here comes the best part. If you have been holding onto your paper so far, now squeeze that paper. Because these techniques are not some ancient methods. Not at all. Both of these methods are from the same year as this technique. The same year? Such improvements in less than a year. That is outstanding. What a time to be alive. Now, I made a promise to you early in the video and the promise was explaining this. Yes, the technique generates results even when not given a lot of instruction. Yes, this was the early example. When we unchecked all the boxes. What quality images can we expect then? Well, here are some uncurated examples. This means that the authors did not cherry pick here, just dumped a bunch of results here and oh my goodness, these are really good. The details are really there. The resolution of the images could be improved, but. But we saw with the previous Gaugan paper that this can improve a great deal in just a couple years or with this paper in less than a year. I would like to send a huge congratulations to the scientists at Nvidia. Bravo. And you fellow scholars just saw how much of an improvement we can get. Just one more paper down the line. So just imagine what the next paper could bring. If you are curious, make sure to subscribe and hit the bell icon to not miss it when the follow up paper appears on two minute papers.

What you see here is a report of this exact paper we have talked about, which was made by weights and biases. I put a link to it in the description.

Make sure to have a look.

I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects.

It really is as good as it gets.

Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get.

A free demo today.

Our thanks to WeightsAndBiases for their long standing support and for helping us make.

Better videos for you. Thanks for watching and for your generous support and I'll see you next time.