A YouTube commentary (LARRUZ channel, June 9, 2023) where Dr. Zsolnai reviews NVIDIA's new system that uses large language models to play Minecraft. The narrator explains the method: the LLM reads textual state information, self-generates a curriculum, writes code to execute actions, and stores successful code as reusable skills. Demonstrated capabilities include building a house, creating a Nether portal, and defeating an Enderman. The video emphasizes a performance claim: the system is reported to be >15× faster than AutoGPT (e.g., reducing ~75 AutoGPT iterations to ~5). It also contains a sponsor segment for Weights & Biases.
Questions this video answers
June 9, 2023
More than 15× faster
AutoGPT: ~75 iterations versus the new technique: ~5 iterations (approximate)
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai after an incredible day of Apple, showcasing a culmination of many, many amazing research papers and also light transport algorithms applied to sound waves. Today we are going to have a look at Nvidia's new invention that can play Minecraft, but not in the way you think. It truly is a sight to behold as it is able to gather resources for itself. Go. And mine catch a fish. But these things I expected. However, what I didn't expect is that it can even engage in more complex tasks like building a base or hunting this adorable little piggy. That kind of breaks my heart. Now it can do three more complex things, including fighting, which I will show you in a moment. But let's have a look at the paper. And immediately when looking at the article I was like, what? Are you sure? You see, this paper claims to use large language models to play Minecraft. How is that even possible? Large language models are typically adept at answering text based questions with text based answers. So how does it play a computer game? Well, it doesn't just play the game, but it plays it incredibly well. How? Well, hold on to your papers, fellow scholars. Because it is able to build a curriculum by itself and for itself. You see, it gets some text information about the world it is in, for instance, the items that it has or what time it is and who is nearby. And based on this information, it starts reasoning by itself. Look, for instance, if we have a wooden pickaxe and some stones, then it is a good time to upgrade to a stone pickaxe. Or if it's nighttime and there is a zombie nearby, it is time to grab a sword and shield. Now these are simple statements, but the key is that this little AI makes these deductions, but by itself we don't need to program these by hand. And thus if it were dropped into a completely different game, it would likely do quite well there too. But wait a minute, this is still just text. How does it become gameplay? Well, after creating this curriculum, the AI writes computer code to control the game and achieve its goals. If this piece of code works, it observes what it has achieved and then stores it away in a skill library. These will be building blocks for new, more complex skills later. So a text based chat assistant that can play a video game, that is very impressive. So how well can it play the game? With a little help in the form of human feedback, it can do three amazing things. One, it can build a house. And I have to say that is some proper craftsmanship there or proper crafts AI ship. Lovely house. Good job little AI. 2. It can also build another portal. This allows it to travel to a different dimension with unique terrain and resources. 3. It can even fight an Enderman. This creature makes short work of an unsuspecting player. However, the AI came well prepared with high quality equipment and bam.
We are done.
Now these all sound good, but we are fellow scholars here so we are looking for a detailed comparison of this technique against previous AIs. Is this any better? Oh my goodness, are you seeing what I am seeing? It can explore so much more than its predecessors from for instance much more than the fan favorite AutoGPT, a version of ChatGPT that can prompt itself and work on its own. And it gets better. Wow, look at that. I can't believe it. It is more than 15 times faster than AutoGPT. AutoGPT has barely come out a few weeks ago and it has already been improved 15x. Just look at that. Wow, what talk. AutoGPT 75 iterations now only takes maybe 5 iterations, likely even less. AutoGPT plateaued at Iron Tools where the new technique is barely getting warmed up. This is incredible improvement in just a couple of months. These text based large language models can not only write code to play video games, but they can even plan their journeys and just imagine what they will be capable of just two more papers down the line. What a time to be alive.
Weights and Biases provides tools to track your experiments in your deep learning projects. What you see here is their amazing sweeps feature which helps you find and reproduce your best runs. And even better, what made this particular run the best. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
