Károly Zsolnai-Fehér walks through Stable Diffusion XL Turbo, an AI that generates images from text at 100 frames per second with no cost. The speed opens possibilities for real-time applications like self-driving cars and urban planning simulations.
Great paper today, fellow scholars. Stable Diffusion XL Turbo. Why? Well, because today we have these amazing computer games and simulations that run quickly and we measure this in frames per second. Then we have offline simulations that run much slower in seconds per frame. And today we have an AI technique that we can measure in case cats per second. You can create so many cats per second. And it can do this too. And surprisingly, it may even help us train self driving cars. We'll look into that. And there's more. You can also try this new tool right now. And we will talk about this amazing technique too. You can also try this for free too. How cool is that? So what was this cat thing? This is Stable Diffusion XL Turbo, a supposedly quicker version of Stable Diffusion, the popular open source text to image AI. The original version can do absolutely amazing things, but it takes a bit about 20 to 60 seconds for an image. And this depends on this setting the number of sampling steps. We typically need 20 to 50 steps to create a high quality image. The more steps, the more computation we have to do and thus the longer we have to wait. And here is an amazing new paper that promises what can that really be? One to four sampling steps, often in a single step. That sounds incredible. I mean if this was true, we would be able to perform text to image in real time. Yes, real time. But wait a second, this is not new. Creating an image in one to four sampling steps has never been a problem. You can do it anytime you want with stable diffusion. But unfortunately then you get this. A blurry image, no detail. So why is this interesting? Dear fellow scholars, this is two minute papers with Dr. Well, it is interesting because this new paper allows you to create images quickly. Quickly, but also at the same time give you high quality images. Now let's have a look at the new technique. Wow, that is as fast as you can type the results, update almost immediately and no more blurry images. Wow. So how quick is it? Well, hold on to your papers, fellow scholars, because it can create an Image in in 9 to 10 milliseconds. Yes, that is 100 cats per second. The resolution is 512 x 512 and the quality is not bad at all. It typically loses only against a slower version of itself, sdxl. But SDXL has been surpassed by a new text to image technique. Yes, we will have a look at that too in a moment. So, quality check mark. But following your prompts closely is also super important. And in that area, check mark too. Excellent. And there is so much more here. It can also perform not only text to image, but image to image translation. One image goes in and it comes out transformed. We have seen this in stable diffusion before and this helps you unleash your creativity like never before. Remember this earlier Nvidia paper where you could draw a landscape and it would almost immediately give you a nearly photorealistic image. Now it can do that too. And not only with landscape images, but with Apple's memojis too. I love the quick iteration speed here. And in fact people are already using it out there in the wild. Let's see how. Here you see an incredible example of real time urban planning, prototyping and visualization. And you can even create animations with it. All this for free and open source. So how is this even possible? It is possible through a technique called adversarial diffusion distillation. Luckily, we have the paper with a detailed description of the phenomenon. So here's how you do it. First, train a complex diffusion model. This starts out from a noisy image. Then over time it learns to reorganize this noise into an image that depicts our text prompt. But it does this slowly. Let's call it the teacher model. Now comes the magic. We now create a smaller student model that tries to mimic its teacher. It learns how the teacher behaves and tries to reproduce its behavior. But wait, we already have the teacher model, so why copy it? Well, we are copying it with the student neural network, so we retain the quality. But at the same time this student network will be much cheaper and faster. So more corgis and cats cheaper and faster. Now hold on to your papers, fellow scholars, because perhaps this could also be used to train self driving cars. How? Well, look at this cool new paper from Nvidia where they use real driving logs to analyze previous situations and even create new ones. Now here all of these agents are controlled by Nvidia's AI. And are you thinking what I am thinking? Oh yes. Just imagine putting this into an image to image translator AI. Two more papers down the line and bam. You have a simulation where you can safely train your cars in challenging situations that actually happened or may happen. Imagine this in a similar manner to this earlier work where we can go from video game graphics to real life and back, but this time with real driving situations. Whoa. And this new paper can tokenize trajectories, meaning that it breaks down complex driving situations, much like you would break down a sentence into words and then letters. And it does it very, very well. How? Well, look, it is able to create more lifelike scenarios, outperforming many many previous techniques. It's like a mini video game with intelligent AI players. What a time to be alive. Now as promised, we are going to have a look at this new text to Image AI that looked at 1.1 billion images and learned to create incredibly high quality outputs for your text prompts. You can try it here. The link is available in the video description. So how good is it? Well let's test it against stable diffusion XL. Look at that. Approximately 6 to 7 times out of 10 it is preferred over SDXL. I think that is insane. Don't forget SDXL is a paper that came out approximately 5 months ago and it has already been surpassed. Bravo. And while we are looking at some of these eye poppingly beautiful images, just imagine the that two more papers down the line and I am sure that we are going to be looking at images and videos of these created in real time. And all you need to provide is just a text prompt experiment tracking, model evaluation and production monitoring for your deep learning projects and LLM apps. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
