A Two Minute Papers video by Dr. Károly Zsolnai-Fehér presents recent AI work that converts segmentation maps into photorealistic video. Earlier methods produced impressive single frames but suffered temporal inconsistency (morphing, color changes). Nvidia’s new technique adds memory and additional inputs (notably depth maps) to enforce consistency and enable neural rendering (novel viewpoints). The presenter also highlights that segmentation input can be cheaply obtained from video games and notes a Lambda GPU Cloud sponsorship.
Chapters
Dear fellow scholars, this is two minute papers with Dr. Approximately three years ago, a magical learning based algorithm appeared that was capable of translating a photorealistic image of a zebra into a horse or the other way around, could transform apples into oranges, and more. Later it became possible to do this even without the presence of a photorealistic image, where all we needed was a segmentation map. This segmentation map provides labels on what should go where. For instance, this should be the road, here will be trees, traffic signs, other vehicles and so on. And the output was a hopefully photorealistic video. And you can see here that the results were absolutely jaw dropping. However, look, as time goes by, the backside of the car morphs and warps over time, creating unrealistic results that are inconsistent even on the short term. In other words, things change around from second to second and the AI does not appear to remember what it did just a moment ago. This kind of consistency was solved surprisingly well in a follow up paper from Nvidia in which an AI would look at the footage of a video game, for instance, Pac man, for approximately 120 hours. We could shut down the video game and the AI would understand the rules so well that it could recreate the game that we could even play with it, had memory and used it well. And therefore it could enforce a notion of world consistency. Or in other words, if we return to a state of the game that we visited before, it will remember to present us with very similar information. So the question naturally arises, would it be possible to create a photorealistic video from these segmentation maps that is also consistent? And in today's paper, researchers at Nvidia propose a new technique called that requests some additional information. For instance, a depth map that provides a little more information on how far different parts of the image are from the camera. Much like the Pac man paper, this also has memory and I wonder if it is able to use it as well as that one did. Let's test it out. This previous work is currently looking at a man with a red shirt. We slowly look away, disregard the warping, and when we go back, hey, do you see what I see here? The shirt became white. This is not because the person is one of those artists who can change their clothes in less than a second, but because this older technique did not have a consistent internal model of the world. And now let's see the new one once again. We start with the red shirt, look away and then, yes, same red to blue gradient. Excellent. So it appears that this new technique also reuses information from previous frames efficiently, it is finally able to create a consistent video with much less morphing and warping. And even better, we have this advantageous consistency property where if we look at something that we looked at before, we will see very similar information there, but there is more. Additionally, it can also generate scenes from new viewpoints which we also refer to as neural rendering. And as you see, the two viewpoints show similar objects. So the consistency property holds here too. And now hold on to your papers because we do not necessarily have to produce these semantic maps ourselves, we can let the machines do all the work by firing up a video game that we like, request that the different object classes are colored differently and get this input for free. And then the technique generated a photorealistic video from the game graphics. Absolutely amazing. Now note that it is not perfect. For instance, it has a different notion of time as the clouds are changing in the background rapidly. And look at the end of the sequence we got back to our starting point and the first frame that we saw is very similar to the last one. The consistency works here too. Very good. I have no doubt that two more papers down the line and this will be even better. And for now we can create consistent photorealistic videos even if all we have is freely obtained video game data. What a time to be alive. If you're a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lamda GPU Cloud I've talked about Lambda's GPU workstations in other videos and am happy to tell you that they are offering GPU cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers because the Lambda GPU cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs. Com papers and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
