This YouTube video (Two Minute Papers, Dr. Zsolnai) describes a learning-based method that converts a single 2D color photograph into an explorable 3D photo. The narrator explains the pipeline: estimating a depth map from a single color image, creating layered geometry, and using image inpainting to fill occluded regions; the result can be rotated on smartphones (using gyroscopes) or viewed in VR. The video also reports runtime measurements (depth estimation ~0.25s, inpainting ~0.5s; full pipeline ~1s) and highlights potential uses including applying the method to historical photos after colorization.
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This is a standard color photo made with a smartphone. Hence it contains only a 2D representation of the world. And when we look at it, our brain is able to reconstruct the 3D information from it. And I wonder, would it be possible for an AI to do the same and go all the way and create a 3D version of this photo that we can rotate around? Well, this new learning based method promises exactly that. And if that is at all possible, even more. These are big words. So let's have a look if it can indeed live up to its promise. So first we take a photograph and we'll find out together in a moment. What kind of phone is needed for this. Probably an amazing one, right? For now this will be the input. And now let's see the 3D photo as an output. Lets rotate this around and. Wow, this is amazing. And you know what is even more amazing? Since pretty much every smartphone is equipped with a gyroscope, these photos can be rotated around in harmony with the rotation of our phones and wait a second, is this some sort of misunderstanding or do I see correctly that we can even look behind a human if we wanted to? That content was not even part of the original photo. How does this work? More on that in a moment. Also just imagine putting on a pair of VR glasses and looking at a plain 2D photo and get an experience as if we were really there. It truly feels like we are living in a science fiction world. If we grab our trusty smartphone and use these images, we can create a timeline full of these 3D photos and and marvel at how beautifully we can scroll such a timeline here and now. We have piled up quite a few questions here. How is this wizardry possible? What kind of phone do we need for this? Do we need a depth sensor? Maybe even lidar? Let's look under the hood and find out together. This is the input. One colored photograph that is expected. And let's continue. Goodness. Now this is unexpected. The algorithm creates a depth map by itself. This depth map tells the algorithm how far different parts of the image are from the camera. Just look at how crisp the outlines are. My goodness, so good. Then with this depth information, it now has an understanding of what is where in this image and creates these layers which is unfortunately not much help. As you remember, we don't have any information on what is behind the person. No matter, because we can use a technique that implements image inpainting to fill in these regions with sensible data. And now with this we can start exploring these 3D photos. So if it created this depth map from the color information, this means that we don't even need a depth sensor for this, just a simple color photograph. But wait a minute. This means that we can plug in any photo from any phone or camera that we or someone else took at any time. And I mean at any time, right? Just imagine taking a black and white photo of a historic event, colorizing it with the previous learning based method and passing this color image to this new method. And then this happens. My goodness. So all this looks and sounds great, but how long do we have to wait for such a 3D photo to be generated? Does my phone battery get completely drained by the time all this computation is done? What is your guess? Please stop the video and leave a comment with your guess. I'll wait. Alright, so is this a battery killer? Lets see. The depth estimation step takes whoa, a quarter of a second in painting. Half a second. And after a little housekeeping we find out that this is not a battery killer at all because the whole process is done in approximately one second. Holy mother of papers. I am very excited to see this technique out there in the wild as soon as possible. What you see here is a report of this exact paper we have talked about which was made by weights and biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. If you work with learning algorithms on a regular basis. Make sure to check out weights and biases. Their system is designed to help you organize your experiments and it is so good it could shave off weeks or even months of work from your projects and is completely free for all individuals, academics and open source projects. This really is as good as it gets and it is hardly a surprise that they are now used by over 200 companies and research institutions. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
