Dr. Zsolnai explains NeRF, a technique that reconstructs 3D scenes from 2D photos using neural networks. Google's NeRF-W improves on this by handling changing light and moving objects, as shown with examples like the Brandenburg Gate and Trevi Fountain. Training takes many hours to days.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Approximately five months ago, we talked about a technique called neural radiance fields, or NERF, in short, that worked on a 5D neural radiance field representation. So what does this mean exactly? What this means is that we have three dimensions for location and two for view direction. Or in short, the input is where we are in space and give it to a neural network to learn it and synthesize new, previously unseen views of not just the materials in the scene, but the entire scene itself. In short, it can learn and reproduce entire real world scenes from only a few views by using neural networks. And the results were just out of this world. Look, it could deal with many kinds of matte and glossy materials and even refractions worked quite well. It also understood depth so accurately that we could use it for augmented reality applications where we put a new virtual object in the scene and it correctly determined whether it is in front of or behind the real objects in the scene. However, not everything was perfect. In many cases, it had trouble with scenes with variable lighting conditions and lots of occluders. You might ask, is that really a problem? Well, imagine a use case of a tourist attraction that a lot of people take photos of, and we then have a collection of photos taken during a different time of the day and of course with a lot of people around. But hey, remember that this means an application where we have exactly these conditions, a wide variety of illumination changes and occluders. This is exactly what NERF was not too good at. Let's see how it did on such a case. Yes, we see both abrupt changes in the illumination and the remnants of the Fox occluding the Brandenburg Gate as well. And this is where this new technique from scientists at Google Research by the name NERF W shines. It takes such a photo collection and tries to reconstruct the whole scene from it, which we can again render from new viewpoints. So how well did the new method do in this case? Let's see. Wow, just look at how consistent those results are. So much improvement in just six months of time. This is unbelievable. This is how it did in a similar case with the Trevi Fountain. Absolutely beautiful. And what is even more beautiful is, is that since it has variation in the viewpoint information, we can change these viewpoints around as the algorithm learned to reconstruct the scene itself. This is something that the original NERF technique could also do. However, what it couldn't do is the same with illumination. Now we can also change the Lighting conditions together with the viewpoint, this truly showcases a deep understanding of illumination and geometry that is not trivial at all. For instance, when loading this scene into this neural re rendering technique from last year, it couldn't tell whether we see just color variations on the same geometry or if the geometry itself is changing. And look, this new technique does much better on cases like this. So clean. Now that we have seen the images, let's see what the numbers say for these scenes. The NRW is the neural re rendering technique we just saw, and the other one is the Nerf paper from this year. The abbreviations show different ways of computing the output images. The up and down arrows show whether they are subject to maximization or minimization. They are both relatively close, but when we look at the new method, we see one of the rare cases where it wins decisively regardless of what we are measuring. Incredible. This paper is truly a great leap in just a few months. But of course not everything is perfect here. The technique may fail to reconstruct regions that are only visible on just a few photos in the input dataset. The training still takes from hours to days. I take this as an interesting detail more than a limitation since the training only has to be done once and then using the technique can take place very quickly. But with that, there you go. A neural algorithm that understands lighting geometry can disentangle the two and reconstruct real world scenes from just a few photos. It truly feels like we are living in a science fiction world. What a time to be alive. What you see here is an instrumentation for a previous paper that we covered in this series which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
