An AI learned to play Minecraft from video and can now generate a playable version in real time. It runs at 20+ FPS and lets you move, place blocks, and light torches, but the visuals are heavily pixelated and it forgets what happened after 3 seconds.
I can't believe this. Now hold on to your papers, fellow scholars, because this AI is able to look at gameplay of Minecraft and learn how the game works. And then it gets crazy after that. It recreates the game for you that you can play in real time right now through the link in the video description. I got to say, it runs like a dream. In a good way and in a bad way too. I really can't believe this because we just talked about Google's project on doing this with Doom and just a few days later, here we are. And these are really amazing because here we don't just give it a text prompt to generate an image or video, but we can interact with the model in real time using a keyboard and mouse. So yes, that means we can walk around, explore, jump and and hid blocks. That is good. But here comes the better part. You can even open up your inventory, place objects, shovel and more. And when placing a torch on the wall, it lights up. It understands that too. And when you eat some food, it kind of removes it from your stockpile if you can make it out from the two pixels that we see. Here now comes the bad part, dear fellow scholars, this is two minute papers with Dr. Zsolnai. Okay, so what's. Well, it is highly pixelated and super low resolution. But not only that, things can get so pixelated. I am not even sure what I am looking at here. This is supposed to be some sort of animal, a pig perhaps. Also, it has memory in the order of a few minutes, right? Not quite. It is less than three seconds. That means no object permanence. Look away, then look back and you are suddenly somewhere else. So it really runs like a dream because they got over 20 frames per second. But it also has this dreamlike quality without remembering what just happened a couple seconds ago. However, that is not the point. The point is that something that was not possible a couple years ago is now suddenly possible. Yes, this might be a historic moment in AI, as this is just the beginning of a new kind of technology. Kind of like what GPT2 was before ChatGPT. GPT2 was interesting, but very few saw the potential in it. And now we have ChatGPT, which is an absolute powerhouse that took the world by storm. Now this is incredibly fast. Here is a weird comparison against previous systems. This does not need 10 to 20 seconds to create one frame. This does over 20 frames per second. Very impressive. Now note that this still runs on proprietary hardware. However, the model weights are available for you brilliant fellow scholars to tinker with and here come the good news as of the making of this video you can try it out yourself through the link in the description. Also this is a small half billion parameter model so it can and likely will be scaled up over time to get higher resolution results. And the specialized hardware they use to run it is something I haven't heard of before that could give a little competition for Nvidia that is always good for the consumer. So make sure to go ahead and try it in the video description and let me know what you think. Have you fellow scholars tried it? How did it go? Let me know in the comments below. And if you feel inspired and you wish to run your own experiments on Nvidia GPUs, check out Lambda. Lambda is a cloud service for large scale AI projects and it is my absolute favorite. Seriously. Try it out now@lambdalabs.com papers or click the link in the description. If you use Lambda you can run your own experiments on super powerful hardware and you also support two minute papers.
