This Evolving AI Finds Bugs in Games | Two Minute Papers #250

Dear fellow scholars, this is two minute papers with Reinforcement learning is a learning algorithm that chooses a set of actions in an environment to maximize a score. This class of techniques enables us to train an AI to master a large variety of video games and has many more cool applications. For instance, in the game of Q Bert, at every time step the AI has to choose the appropriate actions to control this orange character and light up all the cubes without hitting the purple enemy. The this work proposes an interesting alternative to reinforcement learning and is named Evolution Strategies and it aims to train not one agent but an entire population of agents in parallel. The efficiency of this population is assessed much like how evolution works in nature, and new offsprings are created from the best performing candidates. Note that this is not the first paper using evolution strategies. This is a family of techniques that dates back to the 70s. However, an advantage of this variant is that it doesn't require long trial and error sessions to find an appropriate discount factor. But wait, what does this discount factor mean exactly? This is a number that describes whether the AI should focus only on immediate rewards at all costs or whether it should be willing to temporarily make worse decisions for a better payoff in the future. The optimal number is different for every game and depends on how much long term planning it requires. With this evolutionary algorithm we can skip this step entirely. And the really cool thing about this is that it is not only able to master many games, but after only five hours of training it was able to find a way to abuse game mechanics in QBERT in the most creative ways. It has found a glitch where it sacrifices itself to lure the Purple Blob into dropping down after it, and much to our surprise, it found that there is a bug. If it drops down from this position, it should lose a life for doing it, but due to a bug it doesn't. It also learned another cool technique where it waits for the adversary to make a move and immediately goes the other way. Here's the same scene Slowed down, It had also found and exploited another serious bug which was, to the best of my knowledge, previously unknown. After completing the first level, it starts jumping around in a seemingly random manner. A moment later we see that the game does not advance to the next level, but cubes start blinking and the AI is free to score as many points as it wishes. After this video, a human player was able to reproduce this. I've put a link to it in the video description. It also found out the age old trick in Breakout where we dig a tunnel through the bricks, lean back, start reading a paper and let physics solve the rest of the level. One of the greatest advantages of this technique is that instead of training only one agent, it it works on an entire population. These agents can be trained independently, making the algorithm more parallelizable, which means that it is fast and maps really well to modern processors and graphics cards with many cores. And these algorithms are not only winning the game, they are breaking the game. Loving it. What a time to be alive. I think this is an incredible story that everyone needs to hear about. If you wish to help us with our quest and get exclusive perks for this series, please consider supporting us on Patreon. We support we are available through patreon.com twominutepapers and the link with the details is available in the video description. We also use part of these funds to give back to the community and empower research projects and conferences. For instance, we recently sponsored a conference aimed to teach young scientists to write and present their papers at international venues. We are hoping to invest some more into upgrading our video editing rig in the near future. We also support cryptocurrencies such as Bitcoin, Ethereum, and Litecoin. I am really grateful for your support support and this is why every video ends with thanks for watching and for your generous support and I'll see you next time.