New Super Resolution AI - Enhance ~10x Faster!

Én nem gondoltam volna, hogy ezt fizikailag lehetnél. Nézd ezt! Ez egy új szuper rezolúció technika a videók és videójátékéhez, szóval eljön egy nagyon kis pixelált képessége, és utána jön ez az abszolút szép. És az ajánlat, nézd! 270p! Ez egy nagyon nagy rezolúció. Egy kacsa lehet képzelni ezt. So whenever you see LR, that is the low resolution input. Hours marks the new technique, and GT is the ground truth, that is, rendering the true result in a high resolution without any help, which of course takes much longer, but is necessary for comparisons. The Eastern Village scene offers some of the best super resolution results that I have ever seen. Goodness! Wow! It is really close to the ground truth too. And so far, you have seen nothing yet. Look at the medieval dark scene here, hmm, that is some kind of joke, right? I mean, just look at the vegetation here. A bunch of pixels. Would you be able to make a drawing pixel by pixel and guess what this exactly should look like if it were in a higher resolution? For me, not a chance. But this new technique Wow! That is stunning! Yes, it still has trouble with thin structures like this, that is the 2 minute paper special, but to think that it can go from this to this sounds like something straight out of a science fiction movie. And I am really really hoping that this can one day be the future of gaming. Could we just buy a cheapo video card, render the game at a really low resolution, let this AI do its magic, and off we go? Also, this is all well and good, but how fast or slow is it? If you take an hour to do this per image, that is not very practical, is it? So, how long does all this take? Let's have a look together! Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Well, for a previous method from just 3 years ago took more than a hundred milliseconds per frame, which is 10 frames per second just for super resolution. And we haven't even rendered the game, so that is clearly not practical. In fact, it wouldn't fit easily on this chart, because the bars would be so long. Wow, that is almost 10x faster than the technique from 3 years ago, that is stunning. But there are other techniques out there, so what about those? Well, if you are looking for a mild 2x, doubling the resolution, the new one blows the doors off. It can do this 82 times per second. Easily twice, almost 3 times as fast as some of its competitors from last year. Szuper good! Now, let's bump it up to 3x, oh yes, things are getting closer now, but it still comes out ahead, and if we are really demanding we want to quadruple the input resolution, it takes a tiny bit longer than Fuse SR, a fantastic technique from just a year ago. So in that case, I wonder if it is worth waiting this much for. Let's have a look together! Yes, the new one is certainly better here and is worth the longer execution time, while other cases are a bit more subjective. Which one do you feel is better? Let me know in the comments below! And now, let's try to resolve this somehow, together! So far we eyeballed the results, now let's look at the numbers. Meaning let's put a ton of these cases together and calculate a score mathematically. What does mathematics think about subjective cases like this? Who wins? Well, did we get an answer? Yes and no. But mostly yes. Let me try to explain. If we just measure the distance between the pixels compared to the true image, the new technique is clearly better. However, this metric does not understand the image, it just measures distances. If you take an image, and you darken it a bit, or brighten it up, it does not know that it is the same image. Thus, it gives it a poor score. So, they also checked structural similarity That is a bit smarter, as it can take into consideration that a darker version of the image is still the same image. If we look at the overall scores, the new technique does not win everywhere all the time, but it wins most of the cases. I am comfortable handing it the win. Also, technically, this new technique is chef's kiss. Why? Well, get this, it uses the Fourier transform, a mathematical technique for breaking down a signal, for instance, a song, like taking the bass guitar out of a song. And it mixes that with deep neural networks, and hence, this is called deep Fourier-based super-resolution. Oké, so what are the limitations? Is it perfect? Clearly not. Thin structures are not great, there are also cases which are highly subjective. This one is a very typical case of oversharpening vs overblurring. Pick your poison. Temporal splotches are also present, I believe a bit less than for the previous technique from a year ago, but it's still there. That is what we are looking for to make this more practical. And finally, a disadvantage that is relatively easy to isolate. That is fog and particle systems. In those cases, you flat out cannot use this super resolution. Not because it theoretically can't, but mert néz ki egy geometrikus data valamit, amit a G-buffer, és nincs data a partikus szintekben. Ez egy kicsit a büszke, mert ezeket nagyon képes lehetnek simulálni, de teoretikusan lehet kinyitni, úgyhogy, tudod... Csak gondolj, hogy mit lehetünk képzelni. Két több papírt a lényen. That is the first law of papers. For instance, I can imagine that we won't even need the high resolution assets like geometry or textures for a game that we are creating, because we will be able to use super resolution so reliably, that we simply won't need them anymore. Perhaps this will be like compression, but a történetet. Thank you so what do you fellow scholars think? Let me know in the comments below! Úgy látszik.