Dr. Zsolnai covers X fields, a neural method that synthesizes images while independently controlling view direction, lighting, and time. It reconstructs smooth transitions from just five input photos and handles caustics and thin geometry better than prior methods like NeRF.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Approximately five months ago we talked about a technique called neural radiance fields, or nerf. In short, where the input is the location of the camera and an image of what the camera sees. We take a few of those, give them to a neural network to learn them, and synthesize new previously unseen views of not just the materials in the scene, but the entire scene itself. In short, we take a few samples and the neural network learns what should be there. Between the samples, in comes non continuous data, a bunch of photos, and out goes a continuous video where the AI fills in the data between these samples. With this we can change the view direction, but only that. This concept can also be used for other variables. For instance, this work is able to change the lighting, but only the lighting. By the way, this is from long ago, from around two minute papers, episode number 13. So our seasoned fellow scholars know that this was almost 500 episodes ago. Or the third potential variable is time. With this AI based physics simulator, we can advance the time and the algorithm would try to guess how a piece of fluid would evolve over time. This was amazing, but as you might have guessed, we can advance time, but only the time. And this was just a couple of examples from a slew of works that are capable of doing one or at most two of these. These are all amazing techniques, but they offer separate features. One can change the view, but nothing else. One for the illumination, but nothing else, and one for time, but nothing else. With the advent of neural network based learning algorithms, I wonder if it is possible to create an algorithm that does all three. Or is this just science fiction? Well, hold on to your papers because this new work that goes by the name X fields, we can indeed change the time and the view direction and the lighting separately. Or even better, do all three at the same time. Woohoo. Look at how we can play with the time back and forth and set the fluid levels as we desire. That is the time part. And we can also play with the other two parameters as well at the same time. But still, the results that we see here can range from absolutely amazing to to trivial depending on just one factor, and that factor is how much training data was available for the algorithm. Neural networks typically require loads of training data to learn a new concept. For instance, if we wish to teach a neural network what a cat is, we have to show it thousands and thousands of images of cats. So how much training data is needed for this? And now hold on to your papers and whoa, look at These five dots here, do you know what this means? It means that all the AI saw was five images, that is five samples from the scene with different light positions. And it could fill in all the missing details with such accuracy that we can create this smooth and creamy transition. It almost feels like we have made at least 100 photographs of the scene. And all this from five input photos. Absolutely amazing. Now here is my other favorite example. I am a light transport simulation researcher by trade, so by definition, I love caustics. A caustic is a beautiful phenomenon in nature where curved surfaces reflect or refract light and concentrate it to a relatively small area. I hope that you are not surprised when I say that that it is the favorite phenomenon of most light transport researchers. And just look at how beautifully it deals with it. You can take any of these intermediate AI generated images and sell them as real ones, and I doubt anyone would notice. So it does three things that previous techniques could do one by one. But really, how does its quality compare to these previous methods? Let's see how it does on thin geometry, which is a notoriously difficult case for these methods. Here is a previous one. Look. The thick part is reconstructed correctly. However, look at the missing top of the grass blade. Yup, that's gone. A different previous technique by the name local light field fusion not only missed the top as well, but also introduced halo like artifacts to the scene. And as you can see with this footage, the new method solves all of these problems really well. And it is quite close to the true reference footage that we kept hidden from the AI. Perhaps the best part is that it also has an online demo that you can try right now. So make sure to click the link in the video description to have a look. Of course, not even this technique is perfect. There are cases where it might confuse the foreground with the background. And we are still not out of the water when it comes to thin geometry. Also, an extension that I would love to see is changing material properties. Here you see some results from our earlier paper on neural rendering where we can change the material properties of this test object and get a near perfect photorealistic image of it in about 5 milliseconds per image. I would love to see it combined with a technique like this one. And while it looks super challenging, it is easily possible that we will have something like that within two years. The link to our neural rendering paper and its source code is also available in the video description. What a time to be alive. What you see here is a report of this exact paper. We have talked about, which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub, and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through WNB.com papers or or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
