Dr. Zsolnai reviews a neural network that converts video game footage like GTA V into photorealistic images. The new method fixes temporal flicker and material rendering, though it struggles when game scenes don't match the European training data. It runs fast enough for real-time use.
Questions this video answers
June 5, 2021
GTA V
50 real cities
Yes — the presenter says the whole thing runs interactively and can be used as a post-processing step.
Perceptilabs — perceptilabs.com
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This paper is called Enhancing Photorealism Enhancement. Hmm, let's try to unpack what that exactly means. This means that we take video footage from a game. For instance, GTA 5, which is an action game, where the city we can play in was modeled after real places in California. Now, as we're living the advent of neural network based learning algorithms, we have a ton of training data at our disposal on the Internet. For instance, the Cityscapes dataset contains images and videos taken in 50 real cities and it also contains annotations that describe which object is which. And the authors of this paper looked at this and had an absolutely insane idea. And the idea is, let's learn on the Cityscapes dataset what cars, cities and architecture looks like, then take a piece of video footage from the game and translate it into a real movie. So basically something that is impossible, that is an insane idea. And when I read this paper I thought that cannot possibly work in any case, but especially not given that the game takes place in California and the Cityscapes dataset contains mostly footage of German cities. How would a learning algorithm pull that off? There is no way this will work. Now there are previous techniques that attempted this. Here you see a few of them and well, the realism is just not there. And there was an even bigger issue, and that is the lack of temporal coherence. This is the flickering that you see where the AI processes these images independently and does not do that consistently. This quickly breaks the immersion and is typically a deal breaker. And now hold on to your papers and let's have a look together at the new technique. Whoa. This is nothing like the previous ones. It renders the exact same place, the exact same cars. And the badges are still correct and still refer to real world brands. And that's not even the best part. Look. The car paint materials are significantly more realistic. Something that is really difficult to capture in a real time rendering engine. Lots of realistic looking specular highlights off of something that feels like the real geometry of the car. Wow. Now, as you see, most of the generated photorealistic images are dimmer and less saturated than the video game graphics. Why is that? This is because computer game engines often create a more stylized world where the saturation haze and bloom effects are often more pronounced. Lets try to fight this bias where many people consider the more saturated images to be better and focus our attention to the realism in these image pairs. While we are there for reference, we can have a look at what the output would be if we didn't do any of the photorealistic magic, but instead we just tried to breathe more life into the video game footage by trying to transfer the color schemes from these real world videos in the training set. So only color transfer? Let's see. Yes, that helps. Until we compare the results with the photorealistic images synthesized by this new AI. Look, the trees don't look nearly as realistic as the new method. And after we see the real roads, it's hard to settle for the synthetic ones from the game. However, no one said that Cityscapes is the only dataset we can use for this method. In fact, if we still find ourselves yearning for that saturated look, we can try to plug in a more stylized dataset. And get this. This is fantastic because these images don't have many of the limitations of computer graphics rendering systems. Why is that? Because look at the grass here in the game. It looks like a 2D texture. To save resources and be able to render an image quicker, however, the new system can put more real looking grass in there, which is a fully 3D object, where every single blade of grass is considered the most mind blowing thing here is that this AI has finally enough generalization capabilities to learn about cities in Germany and still be able to make convincing photorealistic images for California. The algorithm never saw California, and yet it can recreate it from video game footage better than I ever imagined would be possible. That is mind blowing, unreal. And if you have been holding onto your paper so far, now squeeze that paper. Because here we have one of those rare cases where we squeeze our papers for not a feature, but for a limitation of sorts. You see, there are limits to this technique too. For instance, since the AI was trained on the beautiful lush hills of Germany and Austria, it hasn't really seen the dry hills of la. So what does it do with them? Look, it redrew the hills. The only way its sore hills exist, which is with trees. Now, we can think of this as a limitation, but also as an opportunity. Just imagine the amazing artistic effects we could achieve by playing this trick to our advantage. Also, we don't need to create an 80% photorealistic game like this one and push it up to 100%. With the AI, we could draw not 80%, but the bare minimum, maybe only 20% for the video game. A coarse draft if you will, and let the AI do the heavy lifting. Imagine how much modeling time we could save for artists as well. I love this. What a time to be alive. Now, all of this only makes sense for real world use. If it can run quickly, so can it. How long do we have to wait to get such a photorealistic video? Do we have to wait from minutes to hours? No, the whole thing runs interactively which means that it is already usable. We can plug this into the game as a post processing step. And remember the first law of papers which says that two more papers down the line and it will be even better. What improvements do you expect to happen soon and what would you use this for? Let me know in the comments below. Perceptilabs is a Visual API for TensorFlow, carefully designed to make machine learning as intuitive as possible. This gives you a faster way to build out models with more transparency into how your model is architected, how it performs, and how to debug it. Look, it lets you toggle between the visual modeler and the code editor. It even generates visualizations for all the model variables and gives you recommendations both during modeling and training, and does all this automatically. I only wish I had a tool like this when I was working on my neural networks during my PhD years. Visit perceptilabs.com papers to easily install the free local version of their system today. Our thanks to Perceptilabs for their support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
