YouTube episode (Two Minute Papers) by Dr. Zsolnai that introduces a new AI method which synthesizes photorealistic fly-through videos from a small collection of photos. The host demonstrates that the method produces images that are sharp and closely match withheld ground-truth views, but notes temporal flicker when images are played as video and limitations around thin structures and shiny surfaces. The video emphasizes that training this model requires only a consumer GPU and roughly 12ā24 hours, and includes a sponsorship message from Lambda GPU Cloud advertising Quadro RTX 6000, RTX 8000, and V100 instances (including 48 GB RTX 8000) and a signup link.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a collection of photos like these and magically create a video where we can fly through these photos. How is this even possible? Especially that the input is only a handful of photos? Well, we give it to a learning algorithm and ask it to synthesize a photorealistic video where we fly through the scene as we please. That sounds impossible, especially that some information is given about the scene. But this is really not much. And everything in between these photos has to be synthesized here. Let's see how well this new method can perform that. But don't expect too much. And wow, it took a handful of photos and filled in the rest so well that we got a smooth and creamy video out of it. So the images certainly look good in isolation. Now let's compare it to the real world images that we already have but have hidden from the algorithm and hold on to your papers. Wow, this is breathtaking. It is not a great deal different, is it? Does this mean. Yes, it means that it guesses what reality should look like almost perfectly. Now note that this mainly applies for looking at these images in isolation. As soon as we weave them together into a video and start flying through the scene, we will see some flickering artifacts. But that is to be expected. The AI has to create so much information from from so little. And the tiny inaccuracies that appear in each image are different. And when played abruptly after each other, this introduces these artifacts. So which regions should we look at to find these flaws? Well, usually regions where we have very little information in our set of photos and a lot of variation when we move our head. For instance, visibility around thin structures is still a challenge, but of course you know the joke. How do you spot a two minute paper's viewer? They are always looking behind thin fences. Shiny surfaces are a challenge too, as they reflect their environment and change a lot as we move our head around. So how does it compare to previous methods? Well, it creates images that are sharper and more true to the real images. Look, what you see here is a very rare sight. Usually when we see a new technique like this emerge, it almost always does. Better on some datasets and worse on others. The comparisons are almost always a wash. But here, not at all. Not in the slightest. Look here you see four previous techniques, four scenes and three different ways of measuring the quality of the output images. And almost none of it matters because the new technique reliably outperforms all of them everywhere except here in this one case, depending on how we measure how good a solution is and even then it's quite close. Absolutely amazing. Make sure to also have a look at the paper in the video description to see that it can also perform filmic tone mapping, change the exposure of the output images and more so how did they pull this off? What hardware do we need to train such a neural network? Do we need the server warehouses of Google or OpenAI to make this happen? No, not at all. And here comes the best part. If you have been holding onto your paper so far, now squeeze that paper because all it takes is a consumer graphics card and and 12 to 24 hours of training. Then after that we can use the neural network for as long as we wish. So recreating reality from a handful of photos with the neural network that some people today can train at home themselves, the pace of progress in AI research is absolutely amazing. What a time to be alive.
This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus they are the only cloud service with 48 gigabyte RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
