A short explainer video by Károly Zsolnai-Fehér introducing light field video and 3D photo reconstruction (image inpainting vs. geometry-based reconstruction). It demonstrates capturing scene geometry using many cameras on a spherical shell to enable viewpoint changes during playback, discusses handling reflections and thin geometry, and notes high bandwidth/compression challenges. The video is supported by Lambda GPU Cloud.
In an earlier episode, we explored how to Upgrade these to 3D photos where we could kind of look behind a person. I am saying kind of because what we see here is not reality. This is statistical data that is filled in by an algorithm to match its surroundings, which we refer to as image inpainting. So strictly speaking, it is likely information, but not necessarily true information. And also we can recognize the synthetic parts of the image as they are significantly blurrier. So the question naturally arises in the mind of the curious. How about actually looking behind the person? Is that somehow possible or is that still science fiction? Well, hold on to your papers because this technique shows us the images of the future by sticking a bunch of cameras onto a spherical shell. And when we capture a video, it will see something like this. And the goal is to untangle this mess. And we are not done yet. We also need to reconstruct the geometry of the scene as if the video was captured from many different viewpoints at the same time. Absolutely amazing. And. And yes, this means that we can change our viewpoint while the video is running. Since it is doing the reconstruction in layers. We know how far each object is in these scenes, enabling us to rotate these sparks and flames and look at them in 3D. Yum. Now, I am a light transport researcher by trade, so I hope you can tell that I am very happy about these beautiful volumetric effects. But I would also love to know how it deals with reflective surfaces. Let's see together. Look at the reflections in the sand here. And I'll add a lot of camera movement and wow, this thing works. It really works. And it does not break a sweat even if we try a more reflective surface or an even more reflective surface. This is as reflective as it gets, I'm afraid. And we still get a consistent and crisp image in the mirror. Bravo. Alright, let's get a little more greedy. What about seeing through thin fences? That is quite a challenge. And look at the tail wax there. This is still a touch blurrier here and there, but overall very impressive. So what do we do with a video like this? Well, we can use our mouse to look around within the photo in our web browser. You can try this yourself right now by clicking on the paper in the video description. Make sure to follow the instructions if you do. Or we can make the viewing experience even more immersive with a head mounted display where of course the image will follow wherever we turn our head. Both of these truly feel like entering a photograph and getting a feel of the room they're in. Loving it now since there is a lot of information in these light field videos, it also needs a powerful Internet connection to relay them. Even when using H.265, a powerful video compression standard, we are talking in the order of hundreds of megabits. It is like streaming several videos in 4K resolution at the same time. Compression helps. However, we also have to make sure that we don't compress too much so that compression artifacts don't eat the content behind Thin geometry, or at least not too much. I bet this will be an interesting topic for a follow up paper, so make sure to subscribe and hit the bell icon to not miss it when it appears. And for now, more practical light field photos and videos will be available that allow us to almost feel like we are really in the room with the subjects of the videos. What a time to be alive. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus they are the only cloud service with 48 gigabyte RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lamda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
