An AI agent learns to play Montezuma's Revenge by rewarding itself for surprising discoveries rather than obvious patterns. Using Random Network Distillation, it avoids dead ends like fixating on noise and instead solves the game's puzzle of four keys and six doors, outperforming average humans without any human demonstrations.
Dear fellow scholars, this is two Minute Papers with zsolnai. In a previous episode, we talked about a class of learning algorithms that were endowed with curiosity. This new work also showcases a curious AI that aims to solve Montezuma's Revenge, which is a notoriously difficult platform game for an AI to finish. The main part of the difficulty arises from the fact that the AI needs to be able to plan for longer time periods and and interestingly, it also needs to learn that short term rewards don't necessarily mean long term success. Let's have a look at an example quoting the authors. There are four keys and six doors spread throughout the level. Any of the four keys can open any of the six doors, but are consumed in the process to open the final two doors. The agent must therefore forego opening two of the doors that are easier to find and that would immediately reward it for opening them. So what this means is that we have a tricky situation because the agent would have to disregard the fact that it is getting a nice score from opening the doors and understand that these keys can be saved for later. This is very hard for an AI to resist. And again, curiosity comes to the rescue. Curiosity. At least this particular definition of it works in a way that the harder the guess for the AI what will happen, the more excited it gets to perform an action. This drives the agent to finish the game and explore as much as possible because it is curious to see what the next level holds. You see in the animation here that the big reward spikes show that the AI has found something new and meaningful, like losing a life or narrowly avoiding an adversary. As you also see, climbing a ladder is a predictable, boring mechanic that the AI is not very excited about. Later, it becomes able to predict the results even better the second and third time around. Therefore, it gets even less excited about ladders. This other animation shows how this curious agent explores adjacent rooms over time. This work also introduces a technique that the authors call random network distillation. This means that we start out from a completely randomly initialized, untrained neural network and over time slowly distill it into a trained one. This distillation also makes our neural network immune to the noisy TV problem from our previous episode, where our curious, unassuming agent would get stuck in front of a TV that continually plays new content. It also takes into consideration the score reported by the game and has an internal motivation to explore as well and hold on to your papers. Because it can not only perform well in the game, but this AI is able to perform better than the average human and again, remember that no ground truth knowledge is required. It was never demonstrated to the AI how one should play this game. Very impressive results indeed. And as you see, the pace of progress in machine learning research is nothing short of incredible. Make sure to have a look at the paper in the video description for more details. We'd also like to send a big thank you to Insilico Medicine for supporting this video. They use AI for research on preventing aging, believe it or not, and are doing absolutely amazing work. Make sure to check them out in the video description. Thanks for watching and for your generous support and I'll see you next time.
