A Two Minute Papers video (Dr. Zsolnai) describing Google research called RawNeRF that enables novel-view synthesis from low-light raw sensor images. The speaker contrasts single-image denoising (insufficient) with leveraging multiple raw images (25–200) for improved denoising and NeRF-based view synthesis in the dark, and shows demos: successful reconstruction of specular highlights, thin structures, refocusing/depth-of-field, tone-mapping and exposure control; results significantly outperform prior NeRF methods from ~two years earlier.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a collection of photos like these and magically create a video where we can fly through these photos. And we are going to do all this with a twist. So how is this even possible? Especially that the input is only a handful of photos? Well, typically we give it to a learning algorithm and ask it to synthesize a photorealistic video where we fly through the scene as we please. Of course that sounds impossible, especially that some information is given about the scene. But this is really not much. And as you see, this is not impossible at all. Through the power of learning based techniques, this previous AI is already capable of pulling off this amazing trick. And today I am going to show you something even more incredible. Did you notice that most of these were shot during the daytime? And these are all well lit images, Every single one of them. So our question today is, can we perform view synthesis in the dark? And my initial answer would be a resounding no. Why? Well, in this case we not only have to deal with less detail in the images, it would also be very difficult to stitch new views together if we have images like this one. Luckily we have a choice. Instead we can try something else and that is using raw sensor data instead. It looks like this we get more detail, but uh oh, now we also have a problem. Do you see the problem here? Yes, that's right. In the raw sensor data we have more detail but also much more noise that contaminates this data too. So we either have to choose from less detail and less noise, or from more detail, more noise. So I guess that means that we get no view synthesis in the dark, right? Well don't despair. Not everything is lost yet. There are image denoising techniques that we can reach out to. Let's see if this gets any better. Hmm, it definitely got a lot better. But I have to be honest, this is not even close to the quality we we need for view synthesis. But note that this one denoises a single image. A single image? Yes. Finally there is an opening here. Remember, in the world of nerfs we are not using a single image, we are using a package of images. A package contains much more information than just one image and hopefully it can be denoised better. So this is what the previous method could do. And now hold on to your papers and let's look at this new technique called rawnerf. Can it pull this off? Wow. Seemingly it can. So now can we be greedy and hope that view synthesis Works on this data. Let's see. My goodness. It really does. And we are not done yet. In fact we are just getting started. It can do even more. For instance, it can perform tone mapping on the underlying data to bring out even more detail from these dark images. And here comes my favorite. Oh yes. We can also refocus these images and these highly sought after depth of field effects will start to appear. I love it. And what I love even more is that we can even play with this in real time to refocus the scene. This is a very impressive set of features. So let's take it out for a spin and marvel together at five amazing examples of what it can do. Yes, once again this is extremely noisy. For instance, can you read the street sign? Another chance. Right? And what about now? This looks like magic. I love it. Now let's start the view synthesis part. And this looks really good. Given the noisy inputs, the previous original NERF technique could only produce this. And this is not some ancient technique. No sir. This is from just two years ago and today a couple papers down the line and we get this. I can't believe it. We can even see the specular highlight moving around the badge of the car here. Outstanding. Two actually. Let's have a closer look at specular highlights. Here is a noisy image, the denoised version and the view synthesis and the specular highlights are once again excellent. These are very difficult to capture because they change a great deal as we move the camera around. And the photos are spaced out relatively far from each other. This means a huge challenge for the learning algorithm. And as you see this one passes with flying colors. Three thin structures are always a problem lock an otherwise excellent previews technique. Had a great deal of trouble with the fence here even in a well lit scene. So let's see. Are you kidding me? Doing the same with a bunch of nighttime photos. There is not a chance that this will work. So let's see together. Look at that. I am out of words. Or you know what's even better? Let's be really picky and look here instead. These areas are even more challenging and even these work really well. Such improvement in so little time. For as I am a light transport researcher by trade, I would love to look at it resolve some more challenging specular highlights. For instance, you can see how the road reflects the street lights here. And the result looks not just passable, this looks flat out gorgeous. Now talking about gorgeous scenes, let's look at some more of those five putting it all together. This will be a stress test for the new technique. Let's change the viewpoint, refocus the scene and play with the exposure at the same time. That is incredible. What a time to be alive. And you are saying that it does all this from a collection of 25 to 200 photos today? We can shoot these in seconds. Clearly not even this technique is perfect. We can see that this does not match reality exactly, but going from a set of extremely noisy raw images to this is truly a sight to behold. The previous two year old technique couldn't even get close to these results. Bravo. And this is an excellent place for us to apply. The first law of payload weights and biases provides tools to track your experiments in your deep learning projects. What you see here is their Artifacts feature which speeds up the most common machine learning steps like uploading RAW data, splitting it into training, validation and test sets.
