NVIDIA’s Minecraft AI: Feels Like Magic! 🌴 …Also, 1 Million Subs! 🥳

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. We just hit a million subscribers. I can hardly believe that so many of you fellow scholars are enjoying the papers. Thank you so much for all the love. In the previous episode, we explored an absolutely insane idea. The idea was to unleash a learning algorithm on a dataset that contains images and videos of cities, then take a piece of video footage from a game and translate it into a real movie. It is an absolute miracle that it works. And it not only works, but it works reliably and interactively. It also works much better than its predecessors. Now, we discussed that the input video game footage is pretty detailed, and I was wondering, what if we we don't create the entire game in such detail? What about creating just the bare minimum, a draft of the game, if you will, and let the algorithm do the heavy lifting? Let's call this world to world translation. So is word to word translation possible or is this science fiction? Fortunately, scientists at Nvidia and Cornell University thought of that problem and came up with a remarkable solution. But the first question, what form should this draft take? And they say it should be a Minecraft world. Or in other words, a landscape assembled from little blocks. Yes, that is simple enough indeed. So this goes in, and now let's see what comes out. Oh my. It created water. It understands the concept of an island. And it created a beautiful landscape, also with vegetation insanity. It even seems to have some concept of reflections, although they will need some extra work to get perfectly right. But what about artistic control? Do we get this one solution? Or can we give more detailed instructions to the technique? Yes, we can. Look at that. Since the training data contains desert and snowy landscapes too, it also supports them as outputs. Whoa, this is getting wild. I like it. And it even supports interpolation. Which means that we can create one landscape and ask the AI to create a blend between different styles. We just look at the output animations and pick the one that we like best. Absolutely amazing. What I also really liked is that it also supports rendering fog. But this is not some trivial fog technique. No, no. Look at how beautifully it occludes the trees if we look under the hood. Oh my. I am a light transport researcher by trade, and boy am I happy to see the authors having done their homework. Look, we are not playing games here. The technique contains bona fide volumetric light transmission calculations. Now, this is not the first technique to perform this kind of word to word translation. What about the competition? As you see, there are many prior techniques here, but There is one key issue that almost all of them share. So what is that? Oh yes, much like with the other video game papers, the issue is the lack of temporal coherence, which means that the previous techniques don't remember what they did a few images earlier and may create a drastically different series of images and the result is this kind of flickering that is often a deal breaker regardless of how good the technique is. Otherwise look, the new method does this significantly better. This could help level generation for computer games, creating all kinds of simulations, and if it improves some more, these could maybe even become backdrops to be used in animated movies. Now of course this is still not perfect, some of the outputs are still blocky, but with this method, creating virtual worlds has never been easier. I cannot believe that we can have a learning based algorithm that where the input is one draft world and it transforms it to a much more detailed and beautiful one. Yes, it has its limitations, but just imagine what we will be able to do two more papers down the line, especially given that the quality of the results can be algorithmically measured, which is a godsend for comparing this to future methods. And for now, huge congratulations to Nvidia and Cornell University for this amazing paper. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure. Plus they are the only cloud service with 48 gigabyte of RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.