Researchers combined a slow light field camera with a standard video camera and used neural networks to stitch their outputs together, letting you refocus videos after shooting and shift the viewpoint slightly. The method runs in under 0.1 seconds per frame and could help filmmakers.
Dear fellow scholars, this is two minute papers with Whenever we take an image with our camera and look at it after an event, we often feel that many of them are close to perfect. If only it was less blurry or the focus distance was a bit further away. But the magic moment is now gone and there's nothing to do other than cursing at the blurry footage that we are left with when showing it to our friends. However, if we have access to light fields, we can change some camera parameters after the photo was taken. This includes changing the focal distance or even slightly adjusting the viewpoint of the camera. How cool is that? This can be accomplished by a light field camera, which is also referred to as a planoptic camera. This tries to record not only light intensities, but the direction of incoming light as well. Earlier, this was typically achieved by using an array of cameras. That's both expensive and cumbersome. And here comes the problem with using only one light field camera. Because of the increased amount of data that they have to record, current light field cameras are only able to take three frames per second. That's hardly satisfying if we wish to do this sort of post editing for videos. This work offers a novel technique to remedy this situation by attaching a standard camera to this light field camera. The goal is that the standard camera has 30 so tons of frames per second but with little additional information. And a light field camera which has only a few frames per second but with a ton of additional information. If we stitch all this information together in a smart way, maybe it is a possibility to get full light field editing for videos. Earlier we have talked about interpolation techniques that can fill some of the missing frames in videos. This way we can fill in maybe every other frame in a footage, or we can be a bit more generous than that. However, if we are shown 3 frames per second and we have to create a smooth video by filling the blanks, would almost be like asking an algorithm to create a movie from a comic book. This would be awesome, but we're not there yet. Too much information is missing. This stitching process works with a bit more information than this, and the key idea is to use two convolutional neural networks to fill in the blanks. One is used to predict flows which describe the movements and rotations of the objects in the scene, and one to predict the final appearance of the objects, basically one for how they move and one for how they look. And the results are just absolutely incredible. It is also blazing fast and takes less than a tenth of a second to create one of these new views. Here you can see how the final program is able to change the focal distance of any of the frames in our video, or we can even click on something in the image to get it in focus. And all this is done after the video has been taken. The source code of this project is also available. With some more improvements. This could be tremendously useful in the film industry because the directors could adjust their scenes after the shooting and not just sigh over the inaccuracies and missed opportunities. And this is just one of the many possible other applications. Absolutely amazing. If you have enjoyed this episode, don't forget to subscribe to 2 Minute Papers and also make sure to click the bell icon to never miss an episode. Thanks for watching and for your generous support and I'll see you next time.
Sam.
