This is How Google’s Phone Enhances Your Photos

Dear fellow scholars, this is two minute papers with super resolution is a research field with a ton of published papers every year, where the simplest problem formulation is that we have a low resolution coarse image as an input and we wish to enhance it to get a crisper, higher resolution image. You know, the thing that can always be done immediately and perfectly in many of these detective TV series. And yes, sure, the whole idea of super resolution sounds a little like science fiction. How could I possibly get more content onto an image that's not already there? How would an algorithm know what a blurry text means if it's unreadable? It can't just guess what somebody wrote there, can it? Well, let's see. This paper provides an interesting take on this topic because it rejects the idea of having just one image as an input. You see, in this day and age we have powerful mobile processors in our phones, and when we point our phone camera and take an image, it doesn't just take one, but a series of images. Most people don't know that some of these images are even taken as soon as we open our camera app without even pushing the shoot button. Working with a batch of images is also the basis of the iPhone's beloved Live photo feature. So as a result, this method builds on this raw burst input with multiple images and doesn't need idealized conditions to work properly. Which means that it can process footage that we shoot with our shaky hands. In fact, it forges an advantage out of this imperfection because it can first align these photos and then we have not one image, but a bunch of images with slight changes in viewpoint. This means that we have more information that we can extract from these several images which can be stitched together into one higher quality output image. Now that's an amazing idea if I've ever seen one. It not only acknowledges the limitations of real world usage, but even takes advantage of it. Brilliant. You see throughout this video that the results look heavenly. However, not every kind of motion is desirable. If we have a more complex motion such as the one you see here as we move away from the scene, this can lead to unwanted artifacts in the reconstruction. Luckily, the method is able to detect these cases by building a robustness mask that highlights which are the regions that will likely lead to these unwanted artifacts. Whatever is deemed to be low quality information in this mask is ultimately rejected, leading to high quality outputs, even in the presence of weird motions. And now hold on to your papers. Because this method does not use neural networks or any learning techniques and is orders of magnitude faster than those while providing higher quality images. As a result, the entirety of the process takes only 100 milliseconds to process a really detailed 12 megapixel image, which means that it can do it 10 times every second. These are interactive frame rates, and it seems that doing this in real time is going to be possible within the near future. Huge congratulations to Bart and his team at Google for outmuscling the neural networks. Luckily, higher quality ground truth data can also be easily produced for this project, creating a nice baseline to compare the results to. Here you see that this new method is much closer to this ground truth than previous techniques. As an additional corollary of this solution, the more of these jerky frames we can collect, the better it can reconstruct images in poor lighting conditions, which is typically one of the more desirable features in today's smartphones. In fact, get this this is the method behind Google's Magical Night Sight and Super Res Zoom features that you can access by using their Pixel 3 flagship phones. When this feature came out, I remember that phone reviewers and everyone unaware of the rate of progress in computer graphics research were absolutely floored by the results and could hardly believe their eyes when they first tried it. And I don't blame them. This is a truly incredible piece of work. Make sure to have a look at the paper that contains a ton of comparisons against other methods, and it also shows the relation between the number of collected burst frames and the output quality we can expect as a result. And more. Thanks for watching and for your generous support and I'll see you next time.