Text To Image AIs Just Got Supercharged!
Adobe Firefly 2 makes a major jump over the original version. New research lets you reconstruct real scenes with geometry and lighting so virtual objects blend in realistically, opening up sim2real training and world editing.
Chapters
Three incredible things just happened. One, have a look at Adobe's Firefly Text to image AI. The first version was not bad, it was kinda usable. But Good news, Firefly 2 is already here and it is so much better. Two, we can now control text to image AIs using rich text. This seems like a crazy idea. So. So I wonder what this is for, why is this useful? And three, from now on we will not only visualize, but even alter the world around us. So first, Firefly 2, it can perform text to image. This is where absolute magic happened. Look how version 2 compares to version 1. Not even close. These go from a solid improvement to 2. I am not even sure what is going on here too. My goodness, look. Now that is a huge leap of capabilities for macro images. I love this one. And just imagine what we will be capable of just a couple more papers down the line. I have to be honest, I can't even imagine, can you? Also their gallery of results has also improved leaps and bounds. And don't forget this tool is being handed out to millions and millions of people all around the world in Photoshop and Adobe's other tools. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. But sometimes we don't want to create an entire image. Sometimes we have an image that is just right, but we would like to spice it up with an underground city, waves, new clothes or new planets, or this adorable thing. In these cases, generative fill is going to be excellent. Now Photoshop has been used for decades now to put text with all kinds of options into designs. But here they now introduced something more exciting. Text effects. From now on, you don't even need to be a proper artist to create letters made out of gingerbread, yarn, woven fabric, you name it. They also showcased generative recolor. This is for vector images where the boundaries of the objects are known exactly from a text description. This is a useful feature, but in my opinion this can be done with traditional techniques too. No AI is required. Not too impressive, but a useful addition nonetheless. This video is not sponsored by Adobe. We do not have any business ties with them. This is why I can keep saying that this is good, but this is not that impressive. So all in all, solid techniques, not necessarily the best text to image on the market. We've seen better, but the overall package is good and and it is going to be put in the hands of millions and millions of artists. Now outside of Photoshop, when it comes to text prompts, we have always used plain text. But here is a crazy idea From a really cool paper. So what if we could use rich text effects to make our prompts and as a result make our images more expressive? Sounds great. But wait, is this really useful? I mean, you can add some colorful letters here to the hair and the hair gets the appropriate color, but we could do this anyway. Well, if you are not a believer, check this out. We can also use font styles which would then translate to artistic styles. Hmm, I like it. Now we are getting there. Or if we want to make just one object pop. And there we go. But let's go a bit crazier. Footnotes. This allows us to prescribe more details about an object without having to rewrite the whole prompt. If we don't like it, we change the footnote and there we go. That's a good one too. But let's go even crazier. Links. We can now link to a particular image if we wish to get a particular breed of dog or a particular product for a backpack from our images. And now something really simple but really dreadful. I mean powerful. You'll see in a moment. Let's play with text size. So would you like pineapples on your pizza or would you like pineapples on your pizza? You have to choose one. Choose wisely, fellow scholars. Alright, you know what? Being the good fellow scholars you are, I'll let you get away with more pepperonis or more mushrooms. I would absolutely love to see this in future products. Not the pineapple pizza, but text size. So good. And now hold on to your papers, fellow scholars. Because three, we will now not only visualize but even alter the world around us. First, we capture a few photos of a real world scene and then create a virtual copy of it. So far, so good. Nerfs do that. We have seen lots of works on this already. But here, step number two. The geometry of the scene is then estimated. And then the lighting is estimated too. And now we create a virtual object, then put it into the scene and oh my. That metal basket will actually reflect the light in the scene properly. And. And will cast a beautiful shadow too. And it works elsewhere too. Reflections and shadows appear as they should. What a wonderful paper. And these objects can also be part of a simulation in a virtual world. That really gets my mind going. This way. We could capture a real world place around us and start playing with it as if it were a a video game. Or insert ourselves into a virtual world. Train self driving cars safely within a simulation that is a copy of the real world. Now for this to be really useful, the quality of the results would need to improve a bit, but the concept is now out there. However, this is still an incredibly potent concept. Why? You fellow scholars know very well that one of my favorite phenomena in research is sim2real. What is that? Sim2real is when we would like to train a robot for something in the real world. But when starting out, this little AI is not doing very well at first. It might fail a great deal before it starts to be competent at its job. Take self driving for example, or even a vacuum cleaning robot. Now imagine that we can create a computer simulation and they would train in this simulation safely. And my other favorite, efficiently. You see, one second in real life is one second. However, with a powerful computer, that same one second in real life can be used to simulate hours or even days of progress. If we wait just a bit, we might get a masterful AI that has trained essentially for years and years. And here we add a twist to it. What if the virtual world is based on the real world? We can scan the real world and create a simulated video game from it with that. Now these AIs are able to train in the real world, yet they do it safely and efficiently. And with this new paper add another twist. We have the real world and now we can even alter it. We can create scenarios that would normally be almost impossible in the real world. And the AI will be able to train on it for years and years. I love it. And just imagine what we will be capable of just two more papers down the line. What a time to be alive. Experiment tracking, model evaluation and production monitoring for your deep learning projects and and LLM apps. This is what weights and biases does and it is the best. Everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
