Stable Diffusion 3 generates images with much better text, more creative outputs like fractal humans and kaleidoscopic birds, and realistic reflections. Two technical advances drive it: direct preference optimization for better spelling and human alignment, and rectified flows for more efficient sampling. The 8B-parameter model code and weights are available.
Stable diffusion 3 is a text to image AI where you write a short prompt and you get these beautiful images. And it is, or will soon be a completely open technique that is free for all of us to use. Amazing. But it gets better. Oh yes, the paper is now available. I was lucky to have had access to it a little earlier than most. So here is a deeper look at some of the new results and. And we will talk about how all this wizardry is even possible. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Now results. Three things immediately caught my eye in the paper. One. Text. I am always interested in creating images with text with stable diffusion, but I was never able to do it well enough. Here are some of my results from Stable Diffusion xl, a previous version from just a few months ago. And the results were mixed to say the least. I had to try several times for each text example and at least half of them did not work at all. And even that other half were passable at best. And now let's have a look together at the new technique. Oh my goodness. It not only works, but this is fine. Of course, as soon as we get access, we will see how much cherry picking this requires if, if any. But this looks quite reassuring and it not only seems to work more reliably, but it even supports different styles of text. Loving it. 2. The creativity is incredible. Here are some of my favorites. This is human life depicted out of fractals, some similar mathematical structures, and this was done extremely well. And a kaleidoscopic bird. This one is an absolute beauty. I love how colorfully it pops out from the background. And this translucent pig that has another pig inside it and it just goes on and on and on. Truly excellent. And we get all this for free. Wow. Now three, the quality of these images is also remarkable. For for instance, I did not put this in the creativity, but in the quality category because I think it scores much better here. I love how the jam is dripping into the water, not mixing with it. And as I am a light transport simulation researcher by trade, I cannot help but notice how beautiful the reflections are on the water. My goodness. And then this one, which showcases the third law of papers. Yes, you heard it right. You heard the first law many times. But there are more laws. Four to be exact. The third law says that research is a study of failure. A bad researcher fails a hundred percent of the time, while a good one only fails 99% of the time. Hence, what you see here is always just 1% of the work that was done. This Showcases how much work writing such a paper takes for a group of scientists. Very imaginative. And it is a little ironic too. Love it. But how is this even possible? What does the new technique do to get these incredible results? Well, this is still a diffusion based AI technique that looked at a lot of images and generates new ones by starting out from noise and and over time reorganizing this noise into an image that you desire. Now a few things that caught my eye in the paper. One is a technique called direct preference optimization. If this AI model were a car, this step is essentially a way to fine tune it to get closer to the preferences people typically have when driving a car. Think about a smooth pedal response and for instance, a soft suspension. So does it work? Now hold on to your papers, fellow scholars, and if you feel ready to look at the mangled hand that is coming, you are not ready. But if you think you are, here you go. Oh my goodness. And the text isn't great at all. And with this new step. Lovely. In their user study, scientists found that humans like the new version a great deal better. And it also helps us get spelling a little more reliably. Well, I can't stop thinking about that hand. And I have to agree on this one being better. But there is more to it. Oh yes, rectified flows. Now imagine taking out our fine tuned car for a ride, but on old roads that have lots of twists and turns and the new one is a straight path through the mountains. And rectified flows give us a much better new road that goes straight through the mountains. In other words, it is more sample efficient. Which means that if we give it the same amount of computation time it gets us higher quality results. And all the results that you see here are using the 8 billion parameter network. So many of you will be able to run this on your laptops or use one of the cloud providers to do that. And there will also be a lighter version that might even run on your phone. And all this was lots and lots of work. And we get the fruits of this work for free. Results, code and model weights, all freely available now or soon. Absolutely amazing. Thank you so much. What a time to be alive. And we still have a deeper look at the Gemini 1.5 Pro AI assistant and its free and open model variant Gemma in the works. Subscribe if you wish to hear about those experiment tracking, model evaluation and production monitoring for your deep learning projects and LLM apps. This is what weights and biases does and it is the best everyone is using it. Try it out now. At WNB Me Papers or click the link in the description below.
