A new AI method reconstructs detailed 3D models of people from photos and videos. It captures clothing and geometry from the front, infers the unseen back using image translation, and uses video frames to smooth out flickering. The results beat 2019 approaches.
Dear fellow scholars, this is two minute papers with Dr. Today. A variety of techniques exist that can take an image that contains humans and perform pose estimation on it. This gives us these interesting skeletons that show us the current posture of the subjects shown in these images. Having this skeleton opens up the possibility for many cool applications. For instance, it's great for fall detection and generally many kinds of activity recognition, analyzing athletic performance, and much, much more. But that would require that we can do it not only for still images, but for animations. Can we? Yes, we already can. This is a piece of footage from a previous episode that does exactly that. But what if we wish for more? Let's think bigger. For instance, can we reconstruct not only the pose of the model, but also but the entire 3D geometry of the model itself? You know, including the body shape, face, clothes and more. That sounds like science fiction, right? Or with today's powerful learning algorithms, maybe it is finally a possibility. Who really knows? Let's have a look together and evaluate it with three increasingly more difficult experiments. Let's start with experiment number one. Still images. Nice. I think if I knew these people, I might have a shot at recognizing them solely from the 3D reconstruction. And not only that, but I also see some detail in the clothes. A suit can be recognized and jeans have wrinkles. This new method uses a different geometry representation that enables higher resolution outputs. And it immediately shows check mark. It is clearly working quite well on still images. And now hold on to your papers for experiment number two. Because it can not only deal with still images on the front side only, but it can also reconstruct the backside of the person. Look. My goodness. But hold on for a second. That part of the data is completely unobserved. We haven't seen the backside, so how is that even possible? Well, we have to shift our thinking a little. An intelligent person would be able to infer some of these details. For instance, we know that this is a suit, or that these are boots. And we know roughly what the backside of these objects should look like. This new method leans on an earlier technique by the name image to image translation to estimate this data. And it truly works like magic. If you take a closer look, you see that we have less detail in the backside than the front.
But the fact that we can do.
This is truly a miracle. But we can go even further. I know it is not reasonable to ask, but what about video reconstruction? Let's have a look. Don't expect miracles, at least not yet. There is obviously still quite a bit of flickering left, but the preliminary results are quite encouraging, and I am fairly certain that two more papers down the line and these video results will be nearly as good as the ones for the still images. The key idea here is that the new method performs these reconstructions in a way that is consistent or in other words, if there is a small change in the input model, there will also be a small change in the output model. This is the property that opens up the possibility to extend this method to videos. So how does it compare to previous methods? All of these competing techniques are quite recent as they are from 2019. They appear to be missing a lot of detail and I don't think we would have a chance at recognizing the target subject from the reconstructions. And now, just a year and a half later, look at that incredible progress. It truly feels like we are living in a science fiction world. What a time to be alive. This episode has been supported by Weights and Biases. In this post they show you how.
To use their system to log and.
Visualize these colorful masks for semantic segmentation.
What's more, you can even try an.
Example in an interactive notebook through the link in the video description.
Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub, and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping.
Us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
