Machine learning learns to recreate 2D video games like Super Mario and Mega Man by watching gameplay and predicting the next frame. The AI uses a sprite palette to rebuild the game pixel by pixel, though it's slow and might not work on complex 3D games yet.
Dear fellow scholars, this is two minute papers with in most video games that we've seen for at least a few moments, we learn to anticipate what is going to happen in the next second. And even more, if given the patience and skills, we could attempt to recreate parts of the game itself. And what you see here in this work is actually even better because it requires neither the patience nor the skills to do that.
So.
So here's the million dollar idea. Let's have a learning algorithm look at some video game footage and then ask it to recreate that so we can indulge in playing it. The concept is demonstrated on the Super Mario game and later you will also see some results with the millennial childhood favorite Mega Man. There are many previous works that hook into the source code of these games and try to read and predict what happens next by reading the code level instructions. But not in this case, because this technique looks at the video output and the learning takes place on the level of pixels. Therefore, no access to the inner workings of the game is necessary. The algorithm is given two things for the learning process. One, a sprite palette that contains all the possible elements that can appear in the game, including landscape tiles, enemies, coins, and so on. And two, we also provide an input video sequence with one playthrough of the game to demonstrate the mechanics and possible interactions between these game elements. A video is a series of frames from which the technique learns how a frame can be advanced to the next one. After it has been exposed to enough training samples, it will be able to do this prediction by itself on unknown frames that it hasn't seen before. This pretty much means that we can start playing the game that it tries to mimic. And there are similarities across many games that could be exploited, endowing the learning algorithms with knowledge reused from other games, making them able to recreate even higher quality computer games, even in cases where a given scenario hasn't played out in the training footage. It used to be the privilege of computer graphics researchers to play video games during work hours, but apparently scientists in machine learning also caught up in this regard. Way to go. A word about limitations as the predictions are not very speedy and are based off of a set of facts learned from the video sequences. It is a question as to how well this technique would generalize to more complex 3D video games, as almost all research works. This is a stepping stone, but a very important one at that as this is a proof of concept for a really cool idea. You know the drill a couple papers down the line and we'll see the idea significantly improved. The results are clearly not perfect, but it is a nice demonstration of a new concept. And knowing the rate of progress in machine learning research, you will very soon see some absolutely unreal results. What's even more, I expect that new levels, enemy types and mechanics will soon be synthesized to already existing games via Generative Adversarial Networks or Generative Latent Optimization. If you would like to hear more about these, as always, the links are available in the video description. Also, if you enjoyed this episode, please make sure to help us tell these incredible stories to more and more people by supporting us on Patreon. Your support has always been absolutely amazing. Details are available in the video description. Thanks for watching and for your generous support and I'll see you next time.
Sam.
