Károly Zsolnai-Fehér reviews a neural rendering method from TU Munich and Stanford that learns 3D textures from video to synthesize new viewpoints, rearrange scenes, and create deepfakes. It handles specular highlights and mouth motion better than earlier methods and trains on as few as 125 images, though 1000 images (30 seconds of video) works better.
Questions this video answers
Approximately 4–6 milliseconds.
About 1,000 images (the speaker notes this equals roughly half a minute of video); 125 images produces preliminary results with artifacts.
Chapters
This episode has been supported by Lambda Labs. Dear fellow scholars, this is two minute papers with. In our earlier paper, Gaussian material Synthesis, we made a neural renderer. And what this neural renderer was able to do is reproduce the results of a light transport simulation within 4 to 6 milliseconds in a way that is almost pixel perfect. It took a fixed camera and scene, and we were able to come up with a ton of different materials. And it was always able to guess what the output would look like if we changed the physical properties of a material. This is a perfect setup for material synthesis where these restrictions are not too limiting. Trying to perform high quality neural rendering has been a really important research problem lately, and everyone is asking the question, can we do more with this? Can we move around with the camera and have a neural network predict what the scene would look like? Can we do this with animations? Well, have a look at this new paper, which is a collaboration between researchers at the Technical University of Munich and Stanford University, where all we need is some video footage of a person or object. It takes a close look at this kind of information and can offer three killer applications. One, it can synthesize the object from new viewpoints. Two, it can create a video of this scene and imagine what it would look like if we reorganized it, or can even add more objects to it. And three, perhaps everyone's favorite, performing facial reenactment from a source to a target actor. As with many other methods, these neural textures are stored on top of the 3D objects. However, a more detailed high dimensional description is also stored and learned by this algorithm, which enables it to have a deeper understanding of intricate light transport effects to create these new views. For instance, it is particularly good at reproducing specular highlights, which typically change rapidly as we change our viewpoint for the object. One of the main challenges was building a learning algorithm that can deal with this kind of complexity. The synthesis of mouth movements was always the Achilles heel of these methods. So have a look at how well this one does with it. You can also see with the comparisons here that in general, this new technique smokes the competition. So how much training data do we need to achieve this? I would imagine that this would take hours and hours of video footage, right? No, not at all. This is what the results look like as a function of the amount of training data. On the left, you see that it already kinda works with 125 images, but contains artifacts. But if we can supply a thousand images, we're good. Note that a thousand images sounds like a lot, but it really isn't. It's just half a minute worth of video. How crazy is that? Some limitations still apply. You see one failure case here and the neural network typically needs to be retrained if we wish to use it on new objects, but this worked a few more papers down the line. Respect to Justus for accomplishing this and in general, make sure to have a look at Matthias Niesner's lab which who just got tenured as a full professor and he's only 32 years old. Congratulations. If you have AI related ideas and you would like to try them but not do it in the cloud because you wish to own your own hardware, look no further than Lambda Labs. Lambda Labs offers sleek, beautifully designed laptops, workstations and servers that come pre installed with every major learning framework and updates them for you to taking care of all the dependencies. Look at those beautiful and powerful machines. This way you can spend more of your time with your ideas and don't have to deal with all the software maintenance work. Make sure to go to lambdalabs.compapers or click their link in the video description and look around and if you have any questions you can even call them for advice. Big thanks to Lambda Labs for supporting this video and helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
