Microsoft's New Game AI: How Is This Good?

Scientists at Microsoft just found the holy grail and that is of course finding a way to play video games at work and even get paid for it. So what is going on here? Well, they have a new AI system that looks at footage of people playing video games. And then when we give it a new situation, it tries to predict what is about to happen next. You could call it creating games, sort of. More on that later. Now this is not easy at all, even for these lower resolution videos. Initially when you start training, you get something like this. Note that the resolution of these videos is pretty low, but it is enough so that we can already see the bad news. The footage quickly strays away into something completely different. Now let's train it for a bit longer. Look, now this is a lot better. We are in the same game, the same zone, and jumping is working. We are getting there. And if we train for even longer. Wow. We just noticed now that the previous one was quite wrong because this should have been not jumping, but flying. Also, interaction with objects like this power cell works correctly too. Okay, great. Now our question is, what's the point? What can we use this for? Dear fellow scholars, this is two minute papers with Dr. Well, for instance, you can kind of play with it. You get put into a virt virtual world and when you use your controller to choose different directions, it generates the rest of the level and interactions for you. They call it muddling a wide range of human behaviors. I call it going into three possible directions. Although it is really good progress towards AI based game making. I just want to make sure not to oversell this idea as it stands now. And now comes perhaps the most interesting part. We can not only play with these games once again sort, but we can also change them. For instance, we can put another new object here and the AI will make sure that it stays there and that it is interactable. We can also add a new character to the mix and if we specify it, it will also do something. Now that is really cool. Just imagine having one promo image for a game and this can make it come alive. That is fantastic. But as this is at the moment, would I play with it? Likely not. I'll fire up Civilization or no Man's sky instead. Thank you very much. However, there are two things this might already be really good for. First, rapid prototyping. You can imagine what your game would look like, but ultimately you have to put in a lot of work to make it happen. But with this, not so much. You can very quickly get a feel of the game and then decide whether you wish to work on it or not? Two, if you already have a game, you always want to tinker with something. You have your what if questions. What if we add a barrier here? How would the game change? Did we break something? Can we finish the game at all? These questions can now be explored super super quickly and perhaps automatically. Loving it. Now two more questions remain. One, this is not the only way of using an AI to create a game. There is something that you can do right now. You can ask a chatbot like the new Claude 3.7 to write the code for you for a Snake game that is self aware and even does unexpected things like escaping the Matrix and more. And it gets better. You know the two minute papers special class simulation? Not a problem. I've never seen anything like this done by an AI from scratch. What a beauty. And Grok3 can write a simple flying simulator with multiplayer support. It gets a bit choppy. You know, network code optimization is not trivial if you have hundreds of players flying around in real time. But it is kind of possible. That is mind blowing. So this is the two opposing schools for AI game making bottom up, starting from zero and writing the code. Or top down, learning from video and generating new ones. Which one is the way to go? Well, both are in their early days, but the first law of papers teaches us not to underestimate the pace of progress in AI research. You see, just a couple years ago this was the state of AI video generation and today this. Whoa. And I expect we are going to look at this pixelated game footage with a smile and reminisce about the good old times when we couldn't do it in full, full HD or even higher resolution. Except that I think it will happen not two years from now, possibly in less than a year now. Second question, can this create fundamentally new games? Something that goes beyond the kinds of games it was trained on? I would like to take a guess based on what AI can do in other areas. For instance, it can solve new previously unseen problems in the Mathematical Olympiad. With that in mind, I would say fundamentally new games is a question mark for the next few years. But interesting new variants of already existing concepts that will surely be possible two more papers down the line. That is the power of the papers. I am really looking forward to a future where we can all make our own games like Civilization or no Man's sky right at home, just by talking. And all this in real time. What a time to be alive. This is an incredible paper in the prestigious Nature Journal. Open Access. Free for everyone. Thank you and huge congratulations once again. Very few people are talking about this paper, so I really wanted to show it to you. Subscribe and hit the bell if you wish to see more. And as always, let me know in the comments below what you fellow scholars think. Would you like to run your own copy of Deep Seek in the cloud cheaply without using the official app? Yes. Then try Lambda GPU Cloud. They have so many powerful Nvidia GPUs with tons of memory. I use them regularly too. Seriously, try it out now@lambdalabs.com papers or click the link in the description.