A Two Minute Papers YouTube episode (host: Dr. Károly Zsolnai-Fehér) that presents a new handcrafted technique to synthesize fly-through videos from a small set of photos without training neural networks. The presenter emphasizes extreme speed (examples claim results in ~8.8 minutes or under 20 minutes versus ~1.5 days for prior learning-based methods), slightly better quantitative quality (+0.3 dB), and robustness across forward-facing scenes, full 360° rotations, and foreground/background decomposition. The video ends with a short sponsor mention for Perceptilabs.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a collection of photos like these and magically create a video where we can fly through these photos. And it gets better. We will be able to do it quickly. And get this, no AI is required. So how is this even possible? Especially that the input is only a handful of photos? Well, typically we give it to a learning algorithm and ask it to synthesize a photorealistic video where we fly through the scene as we please. Of course that sounds impossible, especially that some information is given about the scene, but this is really not that much. And as you see, this is not impossible at all. Through the power of learning based techniques, this previous AI is already capable of pulling off this amazing trick. And today I am going to show you that through this incredible new paper, something like this can even be done at home on our own machines. Now the previously showcased technique and its predecessors are building on gathering training data and training a neural network to pull this off. Here you see the training process of one of them compared to the reference results. Well, it looks like we need to be really patient as this process is quite lengthy and, and for the majority of the time we don't get any usable results for nearly a day into this process. Now hold on to your papers because here comes the twist. What is the twist? Well, these are not reference results. No, no. These are the results from the new technique. Yes, you heard it right. It doesn't require a neural network and thus trains so quickly that it almost immediately looks like the final result while the original technique is still unable to produce anything usable. That is absolutely insane. Ok, so it's quick, real quick, but how good are the results? Well, the previous technique was able to produce this after approximately one and a half days of training. And what about the new technique? All it needs is 8.8, 8.8 watt days, hours. No, no, 8.8 minutes. And the result looks like this not only as good, but even a bit better than what the previous method could do in one and a half days. Whoa. So I mentioned that the results are typically even a bit better, which is quite remarkable. Let's take a closer look. This is the previous technique after more than a day. This is the new method after 18 minutes. Now it says that the new technique is 0.3 decibels better. That does not sound like much, does it? Well, note that the decibel scale is not linear, it is logarithmic. What does this mean? It means this. Look, a small numerical difference in the numbers can mean a big difference in quality and it is really close to the real results. All this after 20 minutes of processing. Bravo. And it does not stop there. The technique is also quite robust. It works well on Forward facing scenes, 360 degree rotations, and we can even use it to disassemble a scene into its foreground and background elements. Note that the previous NERF technique we compared to was published just about a year and a half ago. Such incredible improvement in so little time. Here comes the kicker. All this is possible today with a handcrafted technique. No AI is required. What a time to be alive. Perceptilabs is a Visual API for TensorFlow, carefully designed to make machine learning as intuitive as possible. This gives you a faster way to build out models with more transparency into how your model is architected, how it performs and how to debug it. And it even generates visualizations for all the model variables and gives you recommendations both during modeling and training, and does all this automatically. I only wish I had a tool like this when I was working on my neural networks during my PhD years. Visit perceptilabs.com papers and start using their system for free today. Our thanks to Perceptilabs for their support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
