Finally, Beautiful Virtual Scenes…For Less! ☀️

Dear fellow scholars, this is Two Minute.

Papers with Dr. Zsolnai.

Today we are going to see the incredible piece of progress in research works on making our photos come to life. How do we do that? Well of course through view synthesis. To be more exact, we do that through this amazing technique that is referred to as a NERF variant, which means that it is a learning based algorithm that tries to reproduce real world scenes from only a few views in go a few photos of a scene and it has to be able to synthesize new photorealistic images in between these photos. This is view synthesis. In short, as you see here, it can be done quite well with the previous method. So our question number one, why bother publishing a new research paper on this? And question number two, there are plenty of view synthesis papers sloshing around, so why choose this paper? Well, this new technique is called Deonerf NERF via depth oracles. One of the key contributions here is that it is better at predicting how far things are from our camera. And it also takes less time to evaluate than its predecessors. It still makes thin structures a little murky. Look at the tree here. But otherwise the rest seems close to the reference. So is there value in this? Let's look at the results and see for ourselves. We will compare to the original NERF technique first and the results are comparable. What is going on? Is this not supposed to be better? Do these depth oracles help at all? Why is this just comparable to Nerf? Now hold on to your papers because here comes the key. The output of the two techniques may be comparable, but the input isn't. What does that mean? It means that the new technique was given 50 times less information. Whoa, 50 times less? That's barely anything. And when I read the paper, this was the point where I immediately went from a little disappointed to stunned. 50 times less information and it can still create comparable videos?

Yes, the answer is yes.

The depth oracles really work. And it does not end there. There is more. It was also compared against local light field fusion which is from two years ago and the results are much cleaner. And it is also compared against what? This is the neural basis expansion technique nax. In short, we just showcased it approximately two months ago. And not only an excellent follow up paper appeared in the same year, just a few months apart, but it already compares to the previous work and it outperforms it handily. My goodness. The pace of progress in machine learning research never disappoints. And here comes the best part. If you have been holding onto your paper so far, now squeeze that paper because the new technique not only requires 50 times less information. No, no. It is also nearly 50 times cheaper and faster to train the new neural network and to create the new images. So let's pop the question. Is it real time? Yes, all this runs in real time. Look, if we wish to walk around in a photorealistic virtual scene, normally we would have to write a light simulation program and compute every single image separately where each image would take several minutes to finish. Now we just need to shoot a few rays and the neural network will try to understand the data and give us the rest instantly. What a time to be alive. And interestingly, look, the first author of this paper is Thomas Neff who wrote this variant on Nerf. No man as Tolman I guess. Congratulations. And if you wish to discuss this paper, make sure to drop by on our Discord server. The link is available in the video Description this episode has been supported by.

By Weights and Biases. In this post they show you how to use their tool to check and visualize what your neural network is learning.

And even more importantly, a case study.

On how to find bugs in your system and fix them. During my PhD studies I trained a ton of neural networks which were used in our experiments. However, over time there was just too much data in our repositories and what I am looking for is not data but insight. And that's exactly how Weights and Biases helps you by organizing your experiments. It is used by more than 200 companies and research institutions including OpenAI, Toyota Research, GitHub and more. And get this Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get.

A free demo today.

Our thanks to WeightsAndBiases for their long standing support and for helping us make.

Better videos for you. Thanks for watching and for your generous support and I'll see you next time.