Google researchers trained an AI to understand depth by watching cameras move around frozen people. The model learns parallax from the mannequin dataset, then works on moving people with less flicker than before. It enables effects like selective blur and object removal in video.
Dear fellow scholars, this is two minute papers with this paper is about endowing colored images with depth information, which is typically done through depth maps. Depth maps describe how far parts of the scene are from the camera and are given with a color coding, where the darker the colors are, the further away the objects are. These depth maps can be used to apply a variety of effects to the image then that require knowledge about the depth of the objects within. For instance, selectively defocusing parts of the image, or even removing people and inserting new objects to the scene. If we humans look at an image, we have an intuitive understanding of it and have the knowledge to produce a depth map by pen and paper. However, this would of course be infeasible and would take too long, so. So we would prefer a machine to do it for us instead. But of course machines don't understand the concept of 3D geometry, so they probably cannot help us with this or with the power of machine learning algorithms, can they? This new paper from scientists at Google Research attempts to perform this, but with a twist. The twist is that the learning algorithm is unleashed on a dataset of what they call mannequins, or in other words, real humans are asked to stand around frozen in a variety of different positions while the camera moves around in the scene. The goal is that the algorithm would have a look at these frozen people and take into consideration the parallax of the camera movement. This means that the objects closer to the camera move more than the objects that are further away. And turns out this this kind of knowledge can be exploited, so much so that if we train our AI properly, it will be able to predict the depth maps of people that are moving around, even if it had only seen frozen people before. This is particularly difficult because if we have an animation, we have to make sure that the guesses are consistent across time or else we get these annoying flickering effects that you see here with previous techniques. It is still there with the new method, especially for the background, but the improvement on the human part of the image is truly remarkable. Beyond the removal and insertion techniques we talked about earlier, I am also really excited for this method as it may open up the possibility of creating video versions of these amazing portrait mode images with many of the newer smartphones people have in their pockets. Thanks for watching and for your generous support and I'll see you next time.
