Dr. Zsolnai (Two Minute Papers) reviews recent AI methods for creating digital humans from a small collection of photos, contrasting approaches that produce fine details but poor 3D consistency with methods that achieve 3D-consistent views but lose detail. He highlights a new technique claiming 1-megapixel images that combine high detail and strong 3D consistency, shows examples comparing it to prior works (e.g., StyleNeRF, Steinurf), and is optimistic about rapid improvements. The video also includes sponsorship mention of Weights & Biases and their Gradient Descent podcast.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to use an AI to take a collection of photos and hopefully create a digital human out of them. However, not so fast. For instance, have a look at Nvidia's amazing face generator AI. As you see, it can generate beautiful results and they even show us the illusion of these characters that moving around. However, this does not have a strong concept of 3D information. These are 2D images and 3D consistency is not that great. What is that? Well, look, this person just one frame away is not exactly the same person. For instance, the hair moves around. This means that these images are not art directable or at least not easily as you see here. It has improved a great deal over two years and one more research paper down the line, but it's still not quite there. So fine details checkmark, 3D consistency, not quite there. So let's have a look at solutions that have more 3D consistency. To be able to do that, we can take a collection of photos like these and magically create a video where we can fly through these photos. It is really crazy because this is possible today. For instance, here is Nvidia's method that can be trained to perform this in a matter of seconds. This is remarkable, especially that the input is only a handful of photos. Some information is given about the scene, but this is really not much. So can we have a solution with 3D consistency? Yes, indeed. For these 3D consistency check mark. However, the amount of fine details in these outputs is not that great. Hmm, do you see the pattern here? Depending on which previous method we use, we either get tons of detail but no 3D consistency, or we can get our highly coveted 3D consistency. But then the details are gone. So is the dream dead? No digital humans for us. Well, don't despair and have a look at this new technique that promises 1 megapixel images that are also consistent details and consistency at the same time. When I first saw this paper, I said that I will believe it when I see it. This is Steinurf, a previous technique and we see the problems that we now expect. The hair is not consistent, the earring is flickering a great deal, and there are other issues too. So are the authors of the new paper claiming that they can solve all this? Well, now hold on to your papers and have a look at this. This is the new technique. Oh my goodness. The earring, facial features and the hair stay still as we rotate them and it indeed shows the new angles correctly. I love it. And let's examine this phenomenon in a little more detail. Look at the hair here with Style Nerf, the previous technique. It looks decent but it's not quite there and I really wonder why that is. Let's zoom in and have a closer look. Oh yes, upon closer inspection the hair is a bit of a mess and with the new technique, now that is what I call consistency across the video frames. Smooth hair strands everywhere. So good. And what I absolutely loved about this new work is that it outperforms this previous technique which is from how many years ago? Well, if you have been holding on to your papers, now squeeze that paper because this work is from not even one year ago, just from eight months ago and you already see a meaningful improvement over that. That is insanity. Now of course not even this technique is perfect. I don't know for sure if the hair consistency is perfect here and we are dealing with video compression artifacts or whether this is still not 100% there, but this is truly a great step forward. Also here you see that the images are still not as detailed as the photos, but this seems to be a roadblock to me that is so much easier to solve than 3D consistency. My impression is that with this the most difficult part of the task is already done and just one or two more papers down the line, I am sure we will be seeing even more realistic virtual humans. So can we enter into a virtual world as a digital human from just a collection of photos? Oh yes. Time to meet our beloved ones from afar or meet new people and play some games together. What a time to be alive. So does this get your mind going? What would you use this for? Let me know in the comments below.
This video has been supported by Weights and Biases. They have an amazing podcast by the name Gradient Descent where they interview machine learning experts who discuss how they use learning based algorithms to solve real world problems. They've discussed biology, teaching robots, machine learning in outer space, and a whole lot more. Perfect for a fellow scholar with an open mind. Make sure to visit them through WNB MEGD or just click the link in the video description. Our thanks to WeightsAndBiases for their long standing support and for helping us make.
Better videos for you. Thanks for watching and for your generous support and I'll see you next time.
