Dr. Zsolnai explains how neural networks can learn to see depth from video without labeled training data. Instead of needing paired images and depth maps, the unsupervised approach enforces consistency across different camera angles, forcing the network to figure out depth on its own. The results match supervised methods trained on synthetic data.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When we humans look at an image or a piece of video footage such as this one, we all understand that this is just a 2D projection of the world around us. So much so that if we have the time and patience, we could draw a depth map that describes the distance of each object from the camera. This information is highly useful because we can use it to create real time defocus effects for virtual reality and computer games, or even perform this Ken burns effect in 3D. Or in other words, zoom and pan around in a photograph, but with a beautiful twist. Because in the meantime we can reveal the depth of the image. However, when we show the same images to a machine, all it sees is a bunch of numbers. Fortunately, with the ascendancy of neural network based learning algorithms, we now have a chance to do this reasonably well. For instance, we discussed this depth perception neural network in an earlier episode which was trained using a large number of input output pairs, where the inputs are a bunch of images and the outputs are their corresponding depth maps for the neural network to learn from. The authors implemented this with a random scene generator which creates a bunch of these crazy configurations with a lot of occlusions and computes via simulation the appropriate depth map for them. This is what we call supervised learning because we have all these input output pairs. The solutions are given in the training set to guide the training of the neural network. This is supervised learning machine learning with crutches. We can also use this depth information to enhance the perception of self driving cars. But this application is not like the previous two I just mentioned. It is much more, much harder because in the earlier supervised learning example we have trained a neural network in a simulation and then we also use it later in a computer game, which is of course another simulation. We control all the variables and the environment here. However, self driving cars need to be deployed in the real world. These cars also generate a lot of video footage with their sensors which could be fed back to the neural networks as additional training data if we had the depth maps for them, which of course, unfortunately we don't. And now with this, we have arrived to the concept of unsupervised learning. Unsupervised learning is proper machine learning where no crutches are allowed. We just unleash the algorithm on a bunch of data with no labels and if we do it well, the neural network will learn something useful from it. It is very convenient because any video we have may be used as training data. That would be great but we have a tiny problem, and that tiny problem is that this sounds impossible. Or it may have sounded impossible until this paper appeared. This work promises us no less than unsupervised depth learning from videos. Since this is unsupervised, it means that during training all it sees is unlabeled videos from different viewpoints and somehow figures out a way to create these depth maps from it. So how is this even possible? Well, it is possible by adding just one ingenious idea. The idea is that since we don't have the labels, we can't teach the algorithm how to be right, but instead we can teach it to be consistent. That doesn't sound like much, does it? Well, it makes all the difference. Because if we ask the algorithm to be consistent, the it will find out that a good way to be consistent is to be right. While we are looking at some results to make this clearer, let me add one more real world example that demonstrates how cool this idea is. Imagine that you are a university professor overseeing an exam in mathematics and someone tells you that for one of the problems, most of the students gave the same answer. If this is the case, there is a good chance that this was the right answer. It is not a hundred percent chance that this is the case, but if most of the students have the same answer, it is much more unlikely that they have all failed the same way. There are many different ways to fail, but there is only one way to succeed. Therefore, if there is consistency, often there is success. And this simple but powerful thought leads to far reaching conclusions. Let's have a look at some more results. Woohoo. Now this is something. Let me explain why I am so excited for this. This is the input image and this is the perfect depth map that is concealed from our beloved algorithm and is there for us to be able to evaluate its performance. These are two previous works. Both use crutches. The first was trained via supervised learning by showing it input output image pairs with depth maps and it does reasonably well. While the other one gets even less supervision, a worse crutch if you will. And it came up with this. Now the unsupervised new technique was not given any crutches and came up with this Holy mother of papers. It looks like a somewhat coarser but still very accurate version of the true depth maps. So what do you know? This neural network based method looks at unlabeled videos and finds a way to create depth maps by not trying to be right, but trying to be consistent. This is one of those amazing papers where One simple brilliant idea can change everything and make the impossible possible. What a time to be alive. What you see here is an instrumentation of this depth learning paper we have talked about. This was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Also, Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project you can use their tools for free. It is really as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
