Dr. Zsolnai explains Neural Radiance Fields, a machine learning technique that can synthesize video and stabilize footage by reconstructing 3D scenes from images. NERF lets you move a camera through space that was never actually filmed, creating the illusion of stopping or controlling time.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we get to be a paper historian and witness the amazing progress in machine learning research together and learn what is new in the world of nerfs. But first, what is a nerf? In March of 2020, a paper appeared describing an incredible technique by the name Neural Radiance Fields or nerf. In short, this work enables us to take a bunch of input photos and their locations, learn it, and synthesize new previously unseen views of not just the materials in the scene, but the entire scene itself. And here we are talking not only digital environments, but also real scenes as well. Now that's quite a value proposition, especially given that it also supported refractive and reflective surfaces as well. These are both quite a challenge. However, of course NERF had its limitations. For instance, in many cases it had trouble with scenes with variable lighting conditions and lots of occluders. And to my delight, only five months later, in August of 2020, a follow up paper appeared by the name NERF in the Wild, or NERF W. In short, its specialty was tourist attractions that a lot of people take photos of. And we then have a collection of photos taken during a different time of the day. And of course with a lot of people around. And lots of people of course means lots of occlusions. NERFW improved the original algorithm to excel more in cases like this. A few months later, on 2020, November 25, another follow up paper appeared by the name Deformable Neural Radiance Fields D Nerf. The goal here was to take a selfie video and turn it into a portrait that we can rotate around freely. This is something that the authors call a nerfy. If we take the original NERF technique to perform this, we see that it does not do well at all with moving things. And that's where the new deformable variant remains really shines. And today's paper not only has some nice video results embedded in the front page, but it offers a new take on this problem and offers quote, spacetime view synthesis of dynamic scenes. Whoa, that is amazing. But what does that mean? What does this paper really do? The spacetime view synthesis means that we can record a video of someone doing something, since we are recording movement in time and there is also movement in space, or in other words, the camera is moving, both time and space are moving. And what this can do is one freeze one of those variables, in other words, pretend as if the camera didn't move, or 2 pretend as if time didn't move. Or 3 generate new views of the scene while movement takes place. My favorite is that we can pretend to zoom in and even better zoom out even if the recorded video looked like this. Or we can also make a really choppy family memory smoother and much more enjoyable. So how does this compare to previous methods? There are plenty of Nerf variants around. Is this really any good? Let's find out together. This is the original Nerf. We already know about this and we are not surprised in the slightest that it's not so great on dynamic scenes with a lot of movement. However, we what I am surprised by is that all of these previous techniques are from 2020 and all of them struggle with these cases. These comparisons are not against some ancient technology from 1985. No, no, all of them are from the same year. For instance, this previous work is called Consistent Video Depth Estimation and It is from August 2020. We showcased it in this series and and marveled at all of these amazing augmented reality applications that it offered. The snowing example here was one of my favorites. And today's paper appeared just three months later in November 2020 and the authors still took the time and effort to compare against this work from just three months ago. That is fantastic. As you see, this previous method kind of works on this dog, but the lack of information in some regions is quite apparent. This is still maybe usable, but as soon as we transition into a more dynamic example, what do we get? Well, pandemonium. This is true for all previous methods. I cannot imagine that the new method from just a few months later could deal with this difficult case. And look at that. So much better. It is still not perfect. You see that we have lost some detail, but witnessing this kind of progress in just a few months is truly a sight to behold. It really consistently outperforms all of these techniques from the same year. What a time to be alive. If you, like me, find yourself yearning for more quantitative comparisons. The numbers also show that the two variants of the new proposed technique indeed outpace the competition. And it can even do one more thing. Previous video stabilization techniques were good at taking a shaky input video and creating a smoother output. However, these results often came at the cost of a great deal of cropping. Not this new work. Look at how good it is at stabilization and it does not have to crop all this data. Praise the papers.
This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of aws and and Azure plus they are the only cloud service with 48gb RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make.
Better videos for you. Thanks for watching and for your generous support and I'll see you next time.
