Dr. Zsolnai explains how AI fills in missing color and depth data to create 3D photographs with parallax effects. The new method reduces warping and improves image quality scores by 0.5 to 2.5 points over existing approaches, and you can run it directly in your browser.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. We hear more and more about RGBD images these days. These are photographs that are endowed with depth information which enable us to do many wondrous things. For instance, this method was used to endow self driving cars with depth information and worked reasonably well. And this other one provides depth maps that are so consistent we can even add some AR effects to it. And today's paper is going to show what 3D photography is. However, first we need not only color, but depth information in our images to perform these. You see, phones with depth scanners already exist and even more are coming as soon as this year. But even if you have a device that only gives you 2D color images, don't despair. There is plenty of research on how we can estimate these depth maps, even if we have very limited information. And with proper depth information, we can now create these 3D photographs where we get even more information out of one still image. We can look behind objects and see things that we wouldn't see otherwise. Beautiful parallax effects appear as objects at different distances move different amounts. As we move the camera around, you see that the foreground changes a great deal, the buildings in the background less so, and the hills behind them even less so. These photos truly come alive with this new method. An earlier algorithm, the legendary patchmatch method from more than a decade ago, could perform something that we call image inpainting. Image inpainting means looking at what we see in these images and trying to fill in missing information with data that makes sense. The key difference here is that this new technique uses a learning method and does this Image inpainting in 3D and it not only fills in color, but depth information as well. What a crazy, amazing idea. However, this is not the first method to perform this. So how does it compare to other research works? Let's have a look together. Previous methods have a great deal of warping and distortions on the bathtub here. And if you look at the new method, you see that it is much cleaner. There is still a tiny bit of warping, but it is significantly better. The dog head here with this previous method seems to be bobbing around a great deal, while the other methods also have some problems with it. Look at this too. And if you look at how the new method handles it, it is significantly more stable. And you see that these previous techniques are from just one or two years ago. It is unbelievable how far we have come since. Bravo. So this was a qualitative comparison or in other words, we looked at the results. What about the quantitative differences? What do the numbers say? Look at the PSNR column here. This means the peak signal to noise ratio. This is subject to maximization. As the up arrow denotes here, the higher the better. The Difference is between 1/2 to 2 1/2 points when compared to previous methods, which does not sound like a lot at all. So what happened here? Note that PSNR is not a linear but a logarithmic scale. So so this means that a small numeric difference typically translates to a great deal of difference in the images, even if the numeric difference is just 0.5 points on the PSNR scale. However, if you look at ssim, the structural similarity metric, all of them are quite similar and the previous technique appears to be even winning here. But this was a method that warped the dog head and in the visual comparisons the new method came out similar, significantly better than this. So what is going on here? Well, have a look at this metric LPIPS, which was developed at the UC Berkeley OpenAI and Adobe Research. At the risk of simplifying the situation, this uses a neural network to look at an image and uses its inner representation to decide how close the two images are to each other. Loosely speaking, it kind of thinks about the differences as we humans do and is an excellent tool to compare images. And sure enough, this also concludes that the new method performs best. However, this method is still not perfect. There is some flickering going on behind these fences. The transparency of the glass here isn't perfect, but witnessing this huge leap in the quality of results in such little time is truly a sight to behold. What a time to be alive. I started this series to make people feel how I feel when I read these papers and I really hope that it goes through with this paper. Absolutely amazing. What is even more amazing is that with a tiny bit of technical knowledge you can run the source code in your browser. So make sure to have a look at the link in the video description. Let me know in the comments how it went. What you see here is an instrumentation of this exact paper we have talked about, which was made by weights and biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project you you can use their tools for free, it really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
