An AI trained to seek surprises learns to play Pong, Breakout, and Super Mario by chasing novelty instead of points. But the same curiosity that teaches it games can trap it staring at a flickering TV, unable to look away.
Dear fellow scholars, this is two minute papers with reinforcement. Learning is a learning algorithm that chooses a set of actions in an environment to maximize a score. This class of techniques enables us to train an AI to master video games. Avoiding obstacles with a drone, cleaning up a table with a robot arm, and has many more really cool applications. We use the words score and reward interchangeably and the goal is that over time the agent has to learn to maximize a prescribed reward. So where should the rewards come from? Most techniques work by using extrinsic rewards. Extrinsic rewards are only a half solution as they need to come from somewhere, either from the game in the form of a game score, which simply isn't present in every game. And even if it is present in a game, it is very different. For Atari, Breakout and and for instance a strategy game. Intrinsic rewards are designed to come to the rescue. So the AI would be able to completely ignore the in game score and somehow have some sort of inner motivation to drive an AI to complete a level. But what could possibly be a good intrinsic reward that would work well on a variety of tasks? Shouldn't this be different from problem to problem? If so, we are back to square one. If we are to call our learner intelligent, then we need one algorithm that is able to solve a large number of different problems. If we need to reprogram it for every game, that's just a narrow intelligence. So a key finding of this paper is that we can endow the AI with a very human like property, curiosity. Human babies also explore the world out of curiosity and as a happy side effect, learn a lot of useful skills to navigate in this world later. However, as in our everyday speech, the definition of curiosity is a little nebulous. We have to provide a mathematical definition for it. In this work this is defined as trying to maximize the number of surprises. This will drive the learner to favor actions that lead to unexplored regions and complex dynamics in a game. So how do these curious agents fare? Well, quite good. In Pong, when the agent plays against itself, it will end up in long matches passing the ball between the two pedals. How about bowling? Well, I cannot resist. But quote the authors for this one. The agent learned to play the game better than agents trained to maximize the clipped extrinsic reward directly. We think this is because the agent gets attracted to the difficult to predict flashing of the scoreboard occurring after the strikes. With a little stretch, one could perhaps say that this AI is showing signs of addiction. I wonder how it would do with modern mobile games with loot boxes, but we'll leave that for future work. Now, how about Super Mario? Well, the agent is very curious to see how the levels continue, so it learns all the necessary skills to beat the game. Incredible. However, the more seasoned fellow scholars immediately find that there is a catch. What if we sit down the AI in front of a TV that constantly plays new material? You may think that this is some kind of a joke, but it's not. It is a perfectly valid issue because due to its curiosity, the AI would have to stay there forever and not start exploring the level. This is the good old definition of TV addiction. Talk about human like properties and sure enough, as soon as we turn off the tv, we the agent gets to work immediately. Who would have thought? The paper notes that this challenge needs to be dealt with over time. However, the algorithm was tested on a large variety of problems and it did not come up in practice. And the key insight is that curiosity is not only a great replacement for extrinsic rewards, the two are often aligned, but curiosity in some cases is even superior to that. This is an amazing value proposition for something that we can run on any problem without any additional work. So curious agents that are addicted to flashing score screens and TVs. What a time to be alive. And if you enjoyed this episode and you wish to help us on our quest to inform even more people about these amazing stories, please consider Supporting us on Patreon.com TwoMinutePapers you can pick up cool perks there to keep your paper's addiction in check. As always, there is a link to it and to the paper in the video description. Thanks for watching and for your generous support and I'll see you next time.
