A short explainer (2 Minute Papers) by Dr. Zsolnai about DeepMind-era advances in Deep Reinforcement Learning for Atari games. The video contrasts early pixel-based RL successes (e.g., Breakout tunnel strategy) with hard-exploration games (Montezuma's Revenge, Pitfall), and highlights a newer approach that learns from unlabeled YouTube human gameplay — aligning latent visual representations so similar game states across videos become close, enabling agents to learn exploration behaviors without access to game internals or scores.
Chapters
Dear fellow scholars, this is 2 Minute Papers with Dr. Zsolnai. Between 2013 and 2015, DeepMind worked on an incredible learning algorithm by the name Deep Reinforcement Learning. This technique looked at the pixels of the game, was given a controller and played much like a human would, with the exception that it learned to play some Atari games on a superhuman level. I have tried to train it a few years ago and would like to invite you for a marvelous journey to see what happened when it starts learning to play an old game, Atari Breakout. At first the algorithm loses all of its lives without any signs of intelligent action. If we wait a bit, it becomes better at playing the game, roughly matching the skill level of an adept player. But here's the catch. If we wait for longer, we get something absolutely spectacular. Over time, it learns to play like a pro and finds out that the best way to win the game is digging a tunnel through the bricks and hit them from behind. This technique is a combination of a neural network that processes the visual data that we see on the screen and a reinforcement learner that comes up with the gameplay related decisions. This is an amazing algorithm, a true breakthrough in AI research. A key point in this work was that the problem formulation here enabled us to measure our progress easily. We hit one brick, we get some points, so do a lot of that. Lose a few lives, the game ends. So don't do that easy enough. But there are other exploration based games like Montezuma's Revenge or, or Pitfall that it was not good at. And man, these games are a nightmare for any AI because there is no score or at the very least it's hard to define how well we are doing because there are no scores. It is hard to motivate an AI to do anything at all other than just wander around aimlessly. If no one tells us if we are doing well or not, which way do we go? Explore this space or go to the next one. How do we solve all this? And with that let's discuss the state of play in AIs playing difficult exploration based computer games. And I think you will love to see how far we have come since. First, there is a previous line of work that infused these agents with a very human like property Curiosity. That agent was able to do much, much better at these games and then got addicted to the tv. But that's a different story. Note that the TV problem has been remedied since and this new method attempts to solve hard exploration games by watching YouTube videos of humans playing the game and learning from that. As you see it Just rips through these levels in Montezuma's Revenge and other games too. So I wonder, how does all this magic happen? How did this agent learn to explore? Well, it has three things going for it that really makes this work. One, the skeptical scholar would say that all this takes is just copy pasting what it saw from the human player. Also, imitation learning is not new, which is a point that we will address in a moment. So why bother with this now? Hold on to your papers and observe as it seems noticeably less efficient than the human teacher was. Until we realized that this is not the human player and this is not the AI, but the other way around. Look, it was so observant and took away so much from the human demonstrations that in the end it became even more efficient than its human teacher. Whoa. Absolutely amazing. And while we are here, I would like to dissect this copy paste argument. You see, it has an understanding of the game and does not just copy the human demonstrator. But even if it just copied what it saw, it would not be so easy because the AI only sees images and it has to translate how the images change in response to us pressing buttons on the controller. We might also encounter the same level, but at a different time, and we have to understand how to vanquish an opponent and and how to perform that. Two, nobody hooked the agent into the game information, which is huge. This means that it doesn't know what buttons are pressed on the controller, no internal numbers or game states are given to it, and most importantly, it is also not given the score of the game. We discussed how difficult this makes everything. Unfortunately, this means that there is no easy way out. It really has to understand what it sees and and mine out the relevant information from each of these videos. And as you see, it does that with flying colors. Loving it. And three, it can handle the domain gap. Previous imitation learning methods did not deal with that too well. So what does that mean? Let's look at this latent space together and find out. This is what a latent space looks like if we just embed the pixels that we see in the videos. Don't worry, I'll tell you in a moment what that is. Here. The clusters are nicely clamped up away from each other, so that's probably good, right? Well, in this problem, not so much. The latent space means a place where similar kinds of data are meant to be close to each other. These are snippets of the demonstration videos that the clusters relate to. Let's test that together. Do you think these images are similar. Yes, most of us humans would say that these are quite similar. In fact, they are nearly the same. So is this a good latent space embedding? No, not in the slightest. This data is similar, therefore these should be close to each other. But this previous technique did not recognize that because these images have slightly different colors aspect ratios. This has a text overlay. But we all understand that despite all that, we are looking at the same game through different windows. So does the new technique recognize that? Oh yes. Beautiful. Praise the papers. Similar game states are now close to each other. We can align them properly and therefore we can learn more easily from them. This is one of the reasons why it can play so well. So there you go. These new AI agents can look at how we perform complex exploration games and learn so well from us that in the end they do even better than we do. And now to get them to write some amazing papers for us. Or, you know, two minute papers episodes what a time to be alive. This episode has been supported by Weights and Biases. In this post they show you how to use their tool to check and visualize what your neural network is learning and even more importantly, a case study on how to find bugs in your system and fix them. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and and the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
