Google and Harvard built an AI that creates smooth, photorealistic fly-through videos from just a few photos. It handles moving cameras around objects and produces sharper geometry and materials than prior methods like MIP-NeRF, with better anti-aliasing for cleaner specular highlights.
Questions this video answers
May 29, 2022
Google and Harvard
MIP-NeRF
Weights & Biases (noted as free for individuals, academics, and open-source projects)
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a collection of photos like these and magically create a video where we can fly through these photos. So how is this even possible? Especially that the input is only a handful of photos? Well, typically we give it to a learning algorithm and ask it to synthesize a photorealistic video where we fly through the scene as we please. Of course that sounds impossible, especially that some information is given about the scene, but this is really not that much. And as you see, this is not impossible at all. Through the power of learning based techniques, this previous AI is already capable of of pulling off this amazing trick. And today I am going to show you that through this incredible new paper, with a little expertise, something like this can be done even at home on our own machines.
Why?
Because now research scientists at Google and Harvard also publish their take on this problem. And they promise two fantastic improvements. Improvement number one, unbounded scenes. No more front facing scene with a stationary camera. They say that now we can rotate the camera around the object and their technique will still work. That is a huge deal if it indeed works. We will see. You know what, hold on to your papers and let's see together right now. Wow. This does not look like an AI made video out of a bunch of photos. This looks like reality. My goodness. And of course you are experienced fellow scholars over there, so I bet you are immediately interested in the fidelity of the geometry which truly is a sight to behold. I am a light transport researcher by trade, so I am additionally looking at the specular highlights. This is as close to reality as it gets. The depth map it produces which describe the distance of these objects from the camera. And they are also silky smooth. Outstanding. So is this a one off result or is this a robust technique that works on a variety of scenes? Well, I bet you know the answer by now. But wait, we talked about two promises. Promise number one was the unbounded scenes with the moving camera. What is promise number two? Well, promise number two is free anti aliasing. Oh boy. This is a technique from computer graphics that helps us overcome the these jagged edges that are usually present in lower resolution images. And it works so well. Check out this comparison against a previous work by the name MIP Nerf. And it is just so much better. The new method truly seems to be in a league of its own. We get smoother lines and pretty much every material and every piece of geometry comes out better. And note that this previous method is not some ancient technique. No, no, MIP Nerf is a technique from just a year ago. Bravo Google and Bravo Harvard. And remember, all this comes from just a quick camera scan and the rest of the details is learned and filled in by an AI. And this technique now truly has a remarkable understanding of the world around us. So just using a commodity camera, walking around the scene and creating a digital video game version of it. Absolutely incredible. Sign me up right now. So what do you think? What else could this be useful for? What do you expect to happen? A couple more papers down the line? Please let me know in the comments below. I'd love to hear your thoughts.
What you see here is a report of this exact paper we have talked about which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights.
And Biases is free for all individuals.
Academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get.
A free demo today.
Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
