Dr. Zsolnai explains NeRF, a neural network that reconstructs 3D scenes from 2D photos and can generate new camera angles. It captures glossy surfaces and refraction, handles occlusion correctly, and works fast enough for AR applications without depth sensors.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. About two years ago, we worked on a neural rendering system which would perform light transport on this scene. And guess how it would change? If we would change the material properties of this test object, it was able to closely match the output of a real light simulation program. And it was near instantaneous as it took less than 5 milliseconds in instead of the 40 to 60 seconds the light transport algorithm usually requires. This technique went by the name Gaussian material synthesis and the learned quantities were material properties. But this new paper sets out to learn something more difficult and also more general. We are Talking about a 5D neural radiance field representation. So what does this mean exactly? What this means is that we have the three dimensions for location and two for view direction. Or in short, the input is where we are in space and what we are looking at and the resulting image of this view. So here we take a bunch of this input data, learn it, and synthesize new previously unseen views of not just the materials in the scene, but the entire scene itself. And here we are talking not only digital environments, but also real scenes as well. Now that's quite a value proposition. So let's see if it can live up to this promise. Wow, so good. Love it. But what is it really that we should be looking at? What makes a good output here? The most challenging part is writing an algorithm that is able to reproduce delicate high frequency details while having temporal coherence. So what does that mean? Well, in simpler words, we are looking for sharp and smooth image sequences. Perfectly matte objects are easier to learn here because they look the same from all directions, while glossier, more reflective materials are significantly more difficult because they change a great deal as we move our head around. And this highly variant information is typically not present in the learned input images. If you read the paper, you'll see these referred to as nonlumbersian materials. The paper and the video contains a ton of examples of these view dependent effects to demonstrate that these difficult scenes are handled really well by this technique. Refractions also look great. Now, if we define difficulty as things that change a lot when we change our position or view direction a little, not only the nonlumbersian materials are going to give us headaches, but occlusions can also be challenging as well. For instance, you can see here how well it handles the complex occlusion situation between the ribs of the skeleton here. It also has an understanding of depth. And this depth information is so accurate that we can do these nice augmented reality applications where we put a new virtual object in the scene and it correctly determines whether it is in front of or behind the real objects in the scene. Kind of what these new iPads do with their LIDAR sensors, but without the sensor. As you see, this technique smokes the competition. So what do you know? Entire real world scenes can be reproduced from only a few views by using neural networks and the results are just out of this world. Absolutely amazing. What you see here is an instrumentation of this exact paper we have talked about, which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Also, Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI Tools, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
