A Two Minute Papers YouTube episode by Dr. Károly Zsolnai-Fehér describing recent advances in video depth estimation for augmented reality and related applications. The speaker contrasts prior methods (which produced blurry, flickering, low-detail depth maps) with a new approach called "consistent video depth estimation" that yields smooth, temporally consistent, and detailed depth maps suitable for effects like real-time defocus, 3D Ken Burns, inserting objects, or simulating flooding. The transcript also notes remaining artifacts (flicker, fine-detail errors around fingers and behind heads) and includes a sponsor message from Weights & Biases.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When we humans look at an image or a piece of video footage, we understand the geometry of the objects in there so well that if we had the time and patience, we could draw a depth map that describes the distance of each object from the camera. This goes without saying. However, what does not go without saying is that if we could teach computers to do the same, we we could do incredible things. For instance, this learning based technique creates real time defocus effects for virtual reality and computer games. And this one performs this Ken burns effect in 3D or in other words, zoom and pan around in a photograph, but with a beautiful twist because in the meantime it also reveals the depth of the image. With this data, we can even try to teach self driving cars about depth perception to enhance their ability to navigate around safely. However, if you look here, you see two key one, it is a little blurry and there are lots of fine details that it couldn't resolve and it is flickering. In other words, there are abrupt changes from one image to the next one which shouldn't be there as the objects in the video feed are moving smoothly. Smooth motion should mean smooth depth maps and it is getting there. But it is still not the case here. So I wonder if we could teach a machine to perform this task better. And more importantly, what new wondrous things can we do if we pull this off? This new technique is called consistent video depth estimation and it promises smooth and detailed depth maps that are of much higher quality than what previous works offer. And now hold on to your papers because finally these maps contain enough detail to open up the possibility of adding new objects to the scene or even flood the room with water or add many other really cool video effects. All of these will take the geometry of the existing real world objects, for instance cats into consideration. Very cool. The reason why we need such a consistent technique to pull this off is because if we have this flickering in time that we've seen here, then the depth of different objects suddenly bounces around over time. Even for a stationary object, this means that if one frame, the ball would be in front of the person, when in the next one it would suddenly think that it has to put the ball behind them and then in the next one front again, creating a not only jarring but quite unconvincing animation. What is really remarkable is that due to the consistency of the technique, none of that happens here. Love it. Here are some more results where you can see that the outlines of the objects in the depth map are really crisp and follow the changes really well over time. The snowing example here is one of my favorites and it is really convincing. However, there are still a few spots where we can find some visual artifacts. For instance, as the subject is waving, there is lots of fine high frequency data around the fingers there and if you look at the region behind the head closely you find some more issues. Or you can find that some balls are flickering on the table as we move the camera around. Compare that to previous methods that could not do nearly as good as this and now we have something that is quite satisfactory. I can only imagine how good this will get two more papers down the line and for the meantime we'll be able to run these amazing effects even without having a real depth camera. What a time to be alive. This episode has been supported by Weights and Biases. In this post, Latent Space shows you how reports from Weights and Biases are central to managing their machine learning workflow and how to use their reports to quickly identify and and debug issues with their deep learning models. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
