A short narrated YouTube explainer by Károly Zsolnai-Fehér about Google Brain's physics-based 3D football (soccer) simulation used to test reinforcement learning agents. The video explains why the task is challenging (balancing short-term control and long-term strategy), describes two input modalities for agents (pixels vs internal game state), notes that the source code is available, and highlights that DeepMind's Impala achieved reliable wins against medium and hard handcrafted teams after extensive training (500 million training steps).
Questions this video answers
Károly Zsolnai-Fehér (channel: LARRUZ).
October 5, 2019.
500 million training steps.
Chapters
Dear fellow scholars, this is two minute papers with Reinforcement learning is an important subfield within machine learning research where we teach an agent to choose a set of actions in an environment to maximize a score. This enables these AIs to play Atari games at a superhuman level, control drones, robot arms, or even create self driving cars. A few episodes ago we talked about DeepMind's behavior suite that opened up the possibility of measuring how these AI AIs perform with respect to the seven core capabilities of reinforcement learning algorithms. Among them were how well such an AI performs when being shown a new problem, how well or how much they memorize, how willing they are to explore novel solutions, how well they scale to larger problems and more. In the meantime, the Google Brain research team has also been busy creating a physics based 3D football or for some of you, soccer simulation where we can ask an AI to control one or multiple players in this virtual environment. This is a particularly difficult task because it requires finding a delicate balance between rudimentary short term control tasks like passing and long term strategic planning. In this environment we can also test our reinforcement learning agents against handcrafted rule based teams. For instance, here you can see that DeepMind's Impala algorithm is the only one that can reliably beat the medium and hard handcrafted teams, specifically the one that was run for 500 million training steps. The easy case is tuned to be suitable for single machine research works where the hard case is meant to challenge sophisticated AIs that were trained on a massive array of machines. I like this idea a lot. Another design decision I particularly like here is that these agents can be trained from pixels or internal game state. Okay, so what does that really mean? Training from pixels is easy to understand but very hard to perform. This simply means that the agents see the same content as what we see on the screen. DeepMind's deep reinforcement learning is able to do this by training a neural network to understand what events take place on the screen and and passes, no pun intended, all this event information to a reinforcement learner that is responsible for the strategic gameplay related decisions. Now what about the other one? The internal game state learning means that the algorithm sees a bunch of numbers which relate to quantities within the game, such as the position of all the players and the ball, the current score, and so on. This is typically easier to perform because the AI is given high quality and relevant information and is not burdened with the task of visually parsing the entire scene. For instance, OpenAI's amazing Dota 2 team learned this way of course to maximize impact. The source code for this project is also available. This will not only help researchers to train and test their own reinforcement learning algorithms on a challenging scenario, but they can also extend it and make up their own scenarios. Now note that so far I tried my hardest not to comment on the names of the players and the teams, but my will to resist just ran out. So go Real Bajans. Thanks for watching and for your generous support and I'll see you next time.
