Dr. Zsolnai reviews a NeRF variant that creates photorealistic images of shiny objects like desserts from just a few photos. The method handles reflections and thin structures better than previous versions, runs about 1000 times faster, and includes available source code and a web demo.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to synthesize beautiful and photorealistic shiny objects. This amazing new technique is a NERF variant, which means that it is a learning based algorithm that tries to reproduce real world scenes from only a few views. In goes a few photos of a scene and it has to be able to synthesize new photorealistic images in between these photos. This is view synthesis. In short, as you see here, it can be done quite well with the previous method. So our question number one, why bother publishing a new research paper on this? And question number two, there are plenty of view synthesis papers sloshing around, so why choose this paper? Well first, when we tested the original NERF method, I noted that thin structures are still quite a challenge. Look, we have some issues here. Ok, that is a good opening for a potential follow up paper. And in a moment we'll see how the new method handles them. Can it do any better? In the green close up you see that these structures are blurry and lack definition. And oh, it gets even worse. The red example shows that these structures are completely missing at places. And given that we have zoomed in quite far and this previous technique is from just one year ago, I wonder how much of an improvement can we expect if in just one year. Let's have a look. Wow, that is outstanding. Both problems got solved just like that. But it does more, way more. This new method also boasts being able to measure reflectance coefficients for every single pixel. That sounds great, but what does that mean? What this should mean is that the reflections and specular highlights in its outputs are supposed to be much better. These are view dependent effects, so they are quite easy to find. What you need to look for is things that change when we move the camera around. Let's test it. Remember, the input is just a sparse bunch of photos. Here they are. And the AI is supposed to fill in the the missing data and produce a smooth video with hopefully high quality specular highlights. These are especially difficult to get right because they change a great deal when we move the camera just a little. Yet look, the AI can still deal with them really well. So yes, this looks good, but we are experienced fellow scholars over here, so. So we'll put this method to the test. Let's try a challenging test for rendering reflections by using this piece of ancient technology called a cd. Not the best for data storage these days, but fantastic for testing rendering algorithms. So I would like to see two things. One is the environment reflected in the silvery regions did we get it? Yes. Checkmark. And two. I would like to see the rainbow changing and bending as we move the camera around. Let's see. Oh, look at how beautifully it has done it. I love it. Checkmark. These specular objects are not just a seedy thing as you see here. They are absolutely everywhere around us. Therefore it is important to get it right for view synthesis. And I must say these results are very close to getting good enough. Where we can put on a VR headset and venture into what is essentially a bunch of photos, not even a video, and make it feel like we are really in the room. Absolutely amazing. When I read the paper I thought, well, all that's great, but we probably need to wait forever to get results like this. And now hold on to your papers. Because this technique is not only much better in terms of quality. No, no. It is more than a thousand times faster. A thousand times in one year I tried to hold on to my papers, but I have to admit that I have failed. And now they are flying about. But that's not all. Not even close. Look. It can also decouple the camera movements from this view dependent specular effect. So what does that mean? It is like a thought experiment where we keep the camera stationary and let the shiny things change as if we moved around. The paper also contains source code and a web demo that you can try yourself right now. It reveals that we still have some more to go until the true real images can be reproduced. But my goodness, this is so much progress in just one paper. Absolutely mind blowing. Now this technique also has its limitations beyond not being as good as the real photos. For instance, the reproduction of refracted thin structures, as you see, not so good. But just think about it. The fact that we need to make up these crazy scenes to be able to give it trouble is a true testament to how good this technique is. And all this improvement in just one year. What a time to be alive. This video has been supported by weights and biases. Check out their recent offering Fully Connected. A place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to weights and biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
