Nvidia trained an AI agent on 33 years of Minecraft footage, wiki pages, and Reddit to play the game. It can explore, build, and fight the Ender Dragon. Human evaluators agreed with its solutions 97% of the time.
July 27, 2022.
"33 years" of Minecraft footage (across many videos).
Over 7,000 Wiki pages.
Approximately 97% agreement with the AI solutions.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see whether Nvidia's modern AI system they call Minddojo can learn to play and build things in Minecraft and more. In an earlier episode, we explored a previous project that goes by the name Genkraft, which performed world to world translation. What is that? Well, simple. We give an AI a very rough draft of a virtual world and out comes a much more detailed and beautiful one. This almost seems like science fiction, really. Just dream up something and it makes it happen. And it gets better. It created water, it understands the concept of an island, and it created a beautiful landscape, also with vegetation insanity. It even seems to have some concept of reflections, although they will need some extra work to get it perfectly right. It also supported interpolation, which means that we can create one landscape and ask the AI to create a blend between different styles. We just look at the output animations and pick the one that we like best. And with this, finally, we also have some artistic control over the mood of the results. Absolutely amazing. But then they thought, why stop there? For instance, DeepMind has a long history of using AI systems to master all kinds of games, from chess to go to Starcraft 2. OpenAI has a Dota 2 AI project that is able to challenge a world champion team. So Nvidia thought, why not try their hands at Minecraft? Minecraft is an open world game, one of the most played games in the world, which is kind of like a virtual sandbox. Here you can build things, explore, but really do whatever you wish. It's a sandbox after all. So scientists at Nvidia thought, let. Let's train an AI to play this game and see what happens. Okay, but how? Well, most AI systems of today require lots and lots of training data. Wait a minute, there's no shortage of that out there on the Internet, that's for sure. Let me explain. We can have this little AI sit down and watch hundreds of thousands of Minecraft tutorial videos on YouTube. 33 years of footage in total. My goodness. Then have it read over 7,000 Wiki pages, and then it shall become the ultimate Reddit lurker. Wow, that makes a ton of sense. Just think about it. It can learn practical knowledge from the tutorial videos, encyclopedic knowledge from the wiki, and it can learn what makes the best creations the best that are shared on Reddit. That sounds fantastic on paper, but there is so much to do and to learn about this game. Can an AI really get an understanding of all this. Well, hold on to your papers and let's see together what this AI could learn from all this data. But wait a minute. This is a gamer AI that uses the controller to move around in a virtual world. But how do we instruct it? Do we need to speak robot? If not? Well, it doesn't even understand English text. What do we do with that? Well, remember OpenAI's GPT3, which is a neural network model that has read almost the entirety of the Internet. This one has proper English knowledge, so we plug that in and bam. Now you can read that Wikipedia. And it gets better because we can now even give it text instructions. Now let's see how well it fares through five of my favorite examples. One, we can ask it to explore an ocean monument. I love this one because the text description is sufficiently vague. How could a machine understand what exploring means? Well, humans do. And this learns from humans as there must be tons of tutorial videos out there on exploration. And it seems to me that it was able to learn the concept correctly. Loving it. 2. It can encircle these llamas with a fence. That is excellent. It understands what object it needs to use, where to move, where to look, and that not even a tiny gap is allowed. Very impressive. 3. This will be a more dangerous quest. Now scoop a bucket of lava. Oh my goodness. Be careful there, do not fall. We are getting there. And got it. 4. We can even ask it to build another portal. It builds the correct portal frame. And did it use the correct materials? Does it work? Yes and yes. Good job little AI. And five, it is final boss time. Literally. Oh yes. Someone was brave enough to ask the AI to fight an Ender dragon. Essentially the final boss of the game. And the AI was brave enough to try it. Well, apparently this is a limited example as it does not appear to be charging at the AI. But the AI seems to know what we are looking for and what fighting entails in this game. Starting this game from scratch and building up everything to defeat such a dragon takes a long time horizon and will be an excellent benchmark for the next AI. One more paper down the line. I'd love it to perform this start to end. Make sure to subscribe. If such a paper appears, I will be here to show it to you. While we are looking at more examples of what it could do, I have to note that we have barely scratched the surface here. For instance, it understands text. Yes, but if we attach a speech recognition AI to this agent, we don't even need to write. It can essentially be a little virtual friend. How cool is that? What a time to be alive. Okay, so this is Minddojo, an AI agent that understands English and can learn to navigate these virtual worlds so well. We can give it a task and it would execute it. And not just some simple ones. We are talking a wide variety of complex tasks. Now, all this footage looks great, but how do we know if this is really performing these tasks correctly? Well, if you have been holding on to your papers, now squeeze that paper. Because they had an experienced human evaluator look at these results and agreed with the AI solutions about 97% of the time. Wow, that is an incredible result. And don't forget, Nvidia is amazing at democratizing these works and putting them into the hands of everyone. And this one is an excellent example of that. If you have some programming knowledge, you can give it a try right now. So does this get your mind going? What would you use this for? Let me know in the comments below. Thanks for watching and for your generous support and I'll see you next time.
