Dr. Zsolnai (Two Minute Papers) reviews a Google paper that enables few-shot image synthesis: given roughly three to four photos of a subject, the method can generate novel, high-fidelity images of that same subject in new poses, materials, contexts, and viewpoints. The video contrasts this capability with prior tools (DALL·E2, Stable Diffusion) which can produce variants but often change the subject identity; it highlights five example capabilities—context transfer, material/property modification, artistic re-rendering, role/character changes, and view synthesis—and notes strong reflections and secondary effects in outputs. The presenter positions this as a significant advance for art generation, product design, and creative uses.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today you are going to see how Google's new AI just supercharged art generation again. Yes, this year we are entering the age of AI based art generation. OpenAI's DALL E2 technique is able to take a piece of text from us and generate a stunning image that that matches this description. Stable diffusion, a similar but open source technique is also now available for everyone to use. And the results are so good that artists are already using these around the world to create illustrations for a novel texture synthesis for virtual worlds, product design and more. So, are we done here? Is there nothing else to improve other than the visual fidelity of the results? Well, not quite. Have a look at two of my favorite AI generated images. This scholar who is desperately trying to hold on to his papers. So I am very happy with this image. But imagine if we were creating a comic, we would need more images of this chap doing other things. Can we do that? Well, we are experienced fellow scholars over here, so we know that variant generation comes to the rescue. Well, have a look. Clearly the AI has an understanding of the image and can create somewhat similar images. Let's see, we get someone with a similar beard, similar paper, which is similarly on fire. The fact that the AI can have a look at such an image and create variants is a miracle of science, but not quite what we are looking for. Why is that? Well, of course, this is a new scholar and we are looking for the previous scholar doing other things. Now let's try our Fox Scientist 2 and see if this was maybe just an anomaly. Maybe Dall E2 is just not into our scholarly content. Let's see. Well, once again the results are pretty good. It understood that the huge ears, gloves, lab coat and the tie are important elements of the image. But ultimately this is a different fox scientist in a different style. So no more adventures for these scientists, right? Have we lost them forever? Should we give up? Well, not so fast. Have a look at Google's new technique which promises a solution to this challenging problem. Yes, they promise that if we are able to take about four images of our subject, for instance, this good boy here, it will be able to synthesize completely new images with them. Let's see. Yes, this is the same dog in the Acropolis. That is excellent. However, wait a minute. This photo is very similar to this one. So basically there was a little synthesis going on here, but otherwise changing the background carries the show here. That is not new. We can do that with already existing tools anyway, so is that it. Well, hold on to your papers, because the answer is no. It goes beyond changing backgrounds. It goes so far beyond that that I don't even know where to start. For instance, here is our little doggy, swimming, sleeping in a bucket. And we can even give this good boy a haircut. That is absolutely insane. And all of these new situations were synthesized by using only four input photos. Wow. Now we're talking. Similarly, if we have a pair of stylish sunglasses, we can ask a bear to wear it, make a cool product photo out of it, or put it in front of the Eiffel Tower. And as I am a light transport researcher by trade, I have to note that even secondary effects like the reflection of the glasses here are modeled really well. And so are the reflections here. Loving it. But it can do so much more than this. Here are five of my favorite examples from the paper. One, we can not only put our favorite teapot into different contexts or see it in use, but we can even reimagine an otherwise opaque object and see what it would look like if it were made of a transparent material like glass. I love it. And I bet that product design people will also love it too. Two, we can create art transitions of our test subject here. The input is only three photos of a dog, but the output. The output is priceless. We can commission art renditions from legendary artists of the past, and all this nearly for free. How cool is that? Three, I hope you like this teapot property modification concept, because we are going to push it quite a bit further. For instance, before repainting our car, we can have a closer look at what it would look like. And we can also reimagine our favorite little pets as other animals. Which one is your favorite? Let me know in the comments below. Mine is the hippo. It has to be the hippo. Look at how adorable it is. And if even that is not enough, four, we can also ask the AI to reimagine our little dog as a chef, a nurse, a police dog, and many others. And all of these images are fantastic. And finally, five, it can perform no less than view synthesis too. We know from previous papers that this is quite challenging. And this one only requests four photos of our cat. And of course, if the cat is refusing to turn the right direction, which happens basically every time, well, no matter. The AI can resynthesize an image of it looking into our desired directions. This one looks like something straight out of a science fiction movie. What a time to be alive. And this is a huge step forward. You See, previous techniques typically struggled with this. Even in the best cases, the fidelity of the results suffer. This alarm clock is not the same as our input photo was, and neither is this one. But with the new technique, now we're talking. Bravo Google. And once again, don't forget, the first law of papers is on full display here. This huge leap happened just one more paper down the line. I am truly stunned by these results and I cannot even imagine what we will be able to do. Two more papers down the line. So what do you think? What would you use this for? Let me know in the comments below. This video has been supported by Weights and Biases. Check out their recent offering Fully Connected, a place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
