DeepMind built an AI that generates playable video games from text, images, or sketches. It learns characters, controls, and environment behavior without supervision, though the output is low-resolution and slow. The presenter sees rapid improvements ahead and applications beyond games, like robotics simulation.
Goodness. DeepMind's new work might be one of the best papers of the year. So what is going on here? Well, today we can use AI techniques to generate images from text. Videos from text. But wait, are you thinking what I am thinking? It is great to look at all this, but this is a game I don't just want to look, I want to play. And DeepMind's amazing new paper is about exactly that. Dear fellow scholars, this is Two Minute Papers with Dr. Karo Zsolnai. Now, believe it or not, we already looked at an earlier Nvidia paper that did something like this. So is this a solved problem? What did it do exactly? And what does DeepMind's new work do that's perhaps even better? Well, this is Nvidia's gamegan. Normally, if we wish to write a computer game, we first envision the game in our mind, then we sit down and do the programming. But this paper did this completely differently. It first looked at someone playing the game and then it was able to code up the game so that it not only looks like it, but it also behaves the same way to our key presses. You see it at work here. Yes. This means that we can even play with it. And it learns the internal rules of the game and the graphics just by looking at some gameplay. We don't need access to the source code or the internal workings of the game. As long as we can just look at it, it can learn the rules. And scientists at DeepMind just put out a paper that made my jaw drop. I mean, look at the list of authors. This is essentially a superglue. Wow, I am very excited. So what does this do? Well, it doesn't even need to look at an already existing game because it makes a game from scratch. Oh my. This sounds not like text to image, not even text to video, but text to video game. So here's the promise. In goes a piece of text. The text goes into a text to image AI that produces an image. And now hold on to your papers as we can now start playing with that image. Wow, Just look at that. A fully AI assisted workflow. It recognizes who should be the playable character and which of this is the environment creates the controls for this character, like moving around and jumping. It also learned the parallax effect, so it knows which the foreground and background is, how far away they are, and how quickly they should move compared to each other. Bravo. This is already incredible. But it gets better. So far, the input was eventually an image. And that image can also be a photo from the Real world, you add the photo and out comes a playable game. Yes, we will talk about the fact that this is quite pixelated in a moment. But you see, we don't even necessarily need a photo from the real world. We can also use a sketch. Just draw something and you get a game out of it. My goodness, isn't that the dream? So good. And it does all this watching videos on the Internet. And let's have a look at how it relates to previous techniques. Those required additional information. For instance, they needed to know the buttons that were pressed. But this one, look, this one is completely unsupervised. That means that we don't even need to label the videos and show which is the playable character and what buttons were pressed. Nothing like that. Just chuck in the videos. And now little AI is it is your time to figure it all out. And so it does. Lovely. Now this does not go as quickly as you see here. This runs at one frame per second. And as all of you fellow scholars can see, it is also heavily pixelated. But we are one scholars here, so we are not fooled by the low pixel count. It is not like the super high resolution detailed output like OpenAI Sora can produce. However, I feel that we are currently at a Dall E1 moment of text to game. What does that mean? Well, this is what Dall E1 was capable of creating. And approximately a year later version 2 could do this. It was a huge leap in capabilities. And just imagine what this technique will look like by the time it will go to version three. I cannot wait to have a look at that. Just imagine using this together with the miracles that computer graphics researchers are capable of today. For instance. Yes, what you are seeing here is not reality. This is a completely made up scene in a computer simulation written by hand. And it computes how the water and everything should move. And a ray tracing based technique simulates what this should look like. And now fellow scholars, imagine two more papers down the line, this AI looking at footage like this. And then we would play with that. Oh my goodness, what a time to be alive. I have written a comment paper on this in Nature Physics. It is a very short and an easy and fun read. And it is freely available for all of you fellow scholars. It is about simulations that look like reality. The link is available in the video description and every good paper goes beyond its own scope a little. And it is no different for this new work. Yes, this paper may also help robots that move around in this world. That is fantastic. Because robotics has a huge data problem. And this work might give this research area just what it needs to improve quicker. And it can also learn a bit about deformations. We can play with this bag of chips and it knows what is supposed to happen. So this creates the games of today and it trains the robots of tomorrow. What an excellent paper. Bravo. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud computer. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.
