Károly Zsolnai-Fehér presents a neural network that generates realistic depth-of-field blur for VR in real time. The method runs at 200 fps on low-latency settings or 20 fps with higher quality, and outperforms existing techniques by 5 points on PSNR. The team released the code and training data.
Questions this video answers
Károly Zsolnai-Fehér
January 30, 2019
Approximately 50 milliseconds per image (≈ 20 fps)
Approximately 5 milliseconds per image (≈ 200 fps)
Second-best ≈ 40; new method ≈ 45 (PSNR)
Chapters
Dear fellow scholars, this is two minute papers with if we are to write a sophisticated light simulation program and we write a list of features that we really wish to have, we should definitely keep an eye on defocus effects. This is what it looks like, and in order to do that, our simulation program has to take into consideration the geometry and thickness of the lenses within our virtual camera. And even though it looks absolutely amazing, it is very costly to simulate that properly. This particular technique attempts to do this in real time and for specialized display types, typically ones that are found in head mounted displays for virtual reality applications. So here we go. Due to popular request, a little VR in 2 minute papers in virtual reality. Defocus effects are especially important because they mimic how the human visual system works. Only a tiny region that we are focusing on looks sharp and everything else should be blurry and, but not any kind of blurry. It has to look physically plausible. If we can pull this off just right, we'll get a great and immersive VR experience. The heart of this problem is looking at a 2D image and being able to estimate how far away different objects are from the camera lens. This is a task that is relatively easy for humans because we have an intuitive understanding of depth and geometry. But of course this is no easy task for a machine to accomplish this. Here a convolutional neural network is used and our seasoned fellow scholars know that this means that we need a ton of training data. The input should be a bunch of images and their corresponding depth maps for the neural network to learn from. The authors implemented this with a random scene generator which creates a bunch of these crazy scenes with a lot of occlusions and computes via simulation the appropriate depth map for them. On the right you see these depth maps, or in other words, images that describe to the computer how far away these objects are. The incredible thing is that the neural network was able to learn the concept of occlusions and was able to create super high quality defocus effects. Not only that, but this technique can also be reconfigured to fit different use cases. If we are okay with spending up to 50 milliseconds to render an image which is 20 frames per second, we can, we can get super high quality images. Or if we only have a budget of 5 milliseconds per image, which is 200 frames per second, we can do that and the quality of the outputs degrades just a tiny bit. While we are talking about image quality, let's have a closer look at the paper where we see a ton of comparisons against previous works and of course against the baseline ground truth knowledge. You see two metrics, psnr, which is the peak signal to noise ratio, and ssim, the structural similarity metric. In this case, both are used to measure how close the output of these techniques is to the ground Truth footage. Both are subject to maximization. For instance, here you see that the second best technique has a peak signal to noise ratio of around 40 and this new method scores 45. Well, some may think that's just a 12ish percent difference, right? No. Note that PSNR works on a logarithmic scale, which means that even a tiny difference in numbers translates to a huge difference in terms of visuals. You can see in the close ups that the output of this new method is close to indistinguishable from the ground truth, a neural network that successfully learned the concept of occlusions and depth by looking at random scenes. Bravo. As virtual reality applications are on the rise these days, this technique will be useful to provide a more immersive experience for the users and to make sure that this method sees more widespread use. The authors also made the source code and the training datasets available for everyone free of charge, so make sure to have a look at that and run your own experiments. If you're interested, I'll be doing that. In the meantime, thanks for watching and for your generous support, and I'll see you next time.
