Károly Zsolnai-Fehér explains how adding curiosity to AI agents, measured by prediction error, lets them explore and learn without external rewards. The approach works in Super Mario and Doom, handling much sparser rewards than previous methods.
Dear fellow scholars, this is two minute papers withehir. There are many research projects about teaching an AI to play video games. Well, we have seen some amazing results from DeepMind's Deep Q learning algorithm that performed on a superhuman level on many games, but faltered on others. What really made the difference is the sparsity of rewards and the lack of longer term planning. What this means is that the more often we see the score change on our screen, the faster we know how well we are doing and and change our strategy if needed. For instance, if we make a mistake in Atari Breakout, we lose a life almost immediately. But in a strategy game, a bad decision may come back to haunt us up to an hour after committing it. So what can we do to build an AI that can deal with these cases? So far we have talked about extrinsic rewards that come from the environment. For instance, our score in a video game. And most existing AIs are for all intents and purposes extrinsic score maximizing machines. And this work is about introducing an intrinsic reward by endowing an AI with one of the most human like attributes, curiosity. But hold on right there. How can a machine possibly become curious? Well, curiosity is defined by whatever mathematical definition we attach to it. In this work, Curiosity is defined as the AI's ability to predict the results of its own actions. This is big because it gives the AI tools to preemptively start learning skills that don't seem useful now, but might be useful in the future. In short, this AI is driven to explore. Even if it hasn't been told how well it is doing. It will naturally start exploring levels in Super Mario even without seeing the score. And now comes the great part. This curiosity really teaches the AI to learn new skills. And when we drop it into a new previously unseen level, it will perform much better than a non curious one. When playing Doom, the legendary first person shooter game, it will also start exploring the level and and is able to rapidly solve hard exploration tasks. The comparisons reveal that an AI infused with Curiosity performs significantly better on easier tasks. But the even cooler part is that with Curiosity we can further increase the difficulty of the games and the sparsity of the external rewards and can expect the agent to do well even when previous algorithms failed. This will be able to play much harder games than previous works. And remember, games are only used to demonstrate the concept here. This will be able to do so much more. Love it. Thanks for watching and for your generous support and I'll see you next time.
