TecoGAN: Super Resolution Extraordinaire!

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Let's talk about video super resolution. The problem statement is simple. In goes a course video, the technique analyzes, it guesses what's missing and out comes a detailed video. However, of course reliably solving this problem is anything but simple. When learning based algorithms were not nearly as good as they are today. This problem was mainly handled by handcrafted techniques. But they had their limits. After all, if we don't see something too well, how could we tell what's there? And this is where new learning based methods, especially this one, come into play. This is a hard enough problem for even a still image, yet this technique is able to do it really well even for videos. Let's have a look. The eye color for this character is blurry, but we see that it likely has a greenish bluish color. And if we gave this problem to a human, this human would know that we are talking about the eye of another human. And we know roughly what this should look like in reality. A human would also know that this must be a bridge and finish the picture. But what about computers? The key is that if we have a learning algorithm that looks at the coarse and fine version of the same video, it it will hopefully learn what it takes to create a detailed video when given a poor one, which is exactly what happened here. As you see, we can give it very little information and it was able to add a stunning amount of detail to it. Now of course, super resolution is a highly studied field these days. Therefore it is a requirement for a good paper to compare to quite a few previous works. Let's see how it stacks up against those. Here we are given a blocky image of this garment and this is the reference image that was coarsened to create this input. The reference was carefully hidden from the algorithms and only we have it. Previous works could add some details, but the results were nowhere near as good as the reference. So what about the new method? My goodness, it is very close to the real deal. Previous methods also had trouble resolving the details of this region where the new method again is very close to reality. It is truly amazing how much this technique understands the world around us from just this training set of low and high resolution videos. Now if you have a closer look at the author list, you see that Niels Ture is also there. He is a fluid and smoke person. So I thought there had to be an angle here for smoke simulations. And yep, there we go. To even have a fighting chance of understanding the importance of this sequence. Let's go back to Nielse's earlier works, which is one of the best papers ever written. Wavelet Turbulence. That's a paper from 12 years ago. Now, some of the more seasoned fellow scholars among you know that I bring this paper up every chance I get, but especially now that it connects to this work we are looking at. You see, Wavelet Turbulence was an algorithm that could take a coarse smoke simulation after it has been created and added fine details to it. In fact, so many fine details that creating the equivalently high resolution simulation would have been near impossible at the time. However, it did not work with images. It required knowledge about the inner workings of the simulator. For instance, it would need to know about the velocities and pressures at different points in this simulation. Now, this new method can do something very similar and all it does is just look at the image itself and improves it without even looking into the simulation data. Even though the flaws in the output are quite clear. The fact that it can add fine details to a rapidly moving smoke plume is an incredible feat. If you look at the comparison against Cyclogan, a technique from just three years ago, this is just a few more papers down the line and you see that this has improved significantly. And the new one is also more careful with temporal coherence. Or in other words, there is no flickering arising from solving the adjacent frames in the video differently. Very good. And if we look a few more papers down the line, we may just get a learning based algorithm that does so well at this task that we would be able to rewatch any old footage in super high quality. What a time to be alive. What you see here is an instrumentation for a previous paper that we covered in this series which was made by weights and biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.