Dr. Zsolnai (Two Minute Papers) describes a computer-graphics neural-network technique that takes a single 2D image (with or without a mask) and predicts three outputs: object geometry, scene lighting, and an object texture map. The predicted elements can be combined and rendered from novel camera viewpoints to synthesize new images (including turntable videos) from one photo. The transcript shows visual comparisons (bird and car examples) against a 2018 category-specific mesh reconstruction method, highlights applications in robot depth perception and rapid virtual-world asset generation, notes minor artifacts (e.g., discoloration near the bird's beak) and upbeat expectations for quick progress, and ends with a sponsor message for Lambda GPU Cloud.
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai in computer graphics research. We spend most of our time dealing with images. An image is a bunch of pixels put onto a 2D plane, which is a tiny window into reality. But reality is inherently 3D. This is easy to understand for us because if we look at a flat image, we see the geometric structures that it depicts. If we look at this image, we know that this is not a sticker, but a three dimensional fluid domain. If I would freeze an image and ask a human to imagine rotating around this fluid domain, that human would do a pretty good job at that. However, for a computer algorithm, it would be extremely difficult to extract the 3D structure out from this image. So can we use these shiny new neural network based learning algorithms to accomplish something like this? Well, have a look at this new technique that takes a 2D image as an input and tries to guess three things. The cool thing is that the geometry problem we talked about is just the first one. Beyond that two, it also guesses what the lighting configuration is that leads to an appearance like this. And three, it also produces the texture map for an object as well. This would already be great. But wait, there's more. If we plug all this into a rendering program, we can also specify a camera position. And this position can be different from the one that was used to take this input image. So what does that mean exactly? Well, it means that maybe it can not only reconstruct the geometry, light and texture of the object, but but even put this all together and make a photo of it from a novel viewpoint. Wow. Let's have a look at an example. There is a lot going on in this image, so let me try to explain how to read it. This image is the input photo and the white silhouette image is called a mask, which can either be given with the image or be approximated by already existing methods. This is the reconstructed image by this technique. And then this is a previous method from 2018 by the name category specific mesh reconstruction. And now hold on to your papers because in the second row you see this technique creating images of this bird from different novel viewpoints. How cool is that? Absolutely amazing. Since we can render this bird from any viewpoint, we can even create a turntable video of it. And all this from just one input photo. Let's have a look at another example. Here you see how it puts together the final car rendering in the first column from the individual elements like geometry, texture and lighting. The other comparisons in the paper reveal that this technique is indeed a huge step up from previous works. Now all this sounds great, but what is all this used for? What are some example applications of this 3D object from 2D image thing? Well, techniques like this can be a great deal of help in enhancing the depth perception capabilities of robots. And of course, whenever we would like to build a virtual world, creating a 3D version of something we only have a picture of can get extremely laborious. This could help a great deal with that too. For this application we could quickly get a starting point with some text, free information and get an artist to fill in the fine details. This might get addressed in a follow up paper. And if you are worried about the slight discoloration around the beak area of this bird, do not despair. As we always say, two more papers down the line and this will likely be improved significantly. What a time to be alive. This episode has been supported by Lambda. If you're a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lambda GPU Cloud. I've talked about Lambda's GPU workstations in other videos and I'm happy to tell you that they are offering GPU Cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers. Because the Lambda GPU Cloud costs less than half of AWS and Azure, make sure to go to lambdalabs.compapers and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
