DeepMind trained AI agents to play Capture the Flag in Quake 3 by learning from raw video, with no programmed rules. In a 40-player tournament, the agents won 74% of games against human teams that won 43% to 52%. The agents also held their own when teamed with humans and stayed effective even with slower reaction times.
August 5, 2018.
40 human players.
43%.
52%.
74%.
Dear fellow scholars, this is two minute papers with after having a look at OpenAI's effort to master the Dota 2 game. Of course, we all know that scientists at DeepMind are also hard at work on an AI that beats the Capture the Flag game mode in Quake 3. Quake 3 Arena is an iconic first person shooter game and Capture the Flag is a fun game mode where each team tries to take the other team's flag and carry it to their own base while protecting their own. This game mode requires good aiming skills, map presence, reading the opponents well, and tons of strategy. A nightmare situation for any kind of AI. Not only that, but in this version the map changes from game to game. Therefore the AI has to learn general concepts and be able to pull them off in a variety of different previously unseen conditions. This doesn't seem to be within the realm of possibilities to pull off. The minimaps here always show the location of the players. Each are color coded to blue or red to indicate their teams. Much like humans. These AI agents learned by looking at the video output of the game and have never been told anything about the game or what the rules are. These scientists at DeepMind ran a tournament with 40 human players who were matched up against these agents randomly, both as opponents and teammates. In this tournament, a team of average human players had a win probability of 43%, where a team of strong players won slightly more than half 52% of their games. And now hold on to your papers because the agents were able to win 74% of their games, so the difference between the average and a strong human player's win rate is 9% and the difference between the strongest humans and the AI is more than twice that margin, 22%. This is insanity. And as you see, it barely matters what the size or the layout of the map is or how many teammates there are. The AI's win rate is always remarkably high. These agents showcase many human like behaviors such as staying at their own base to defend it, camping within the opponent's base, or following teammates. This builds on a new architecture by the name for the win ftw. In short, good work folks. Instead of training one agent, it uses a population of agents that train and evolve from each other to make sure that a diverse set of playstyles are discovered. This uses recurrent neural networks. These are neural network variants that are able to learn and produce sequences of data. Here two of these are used, a fast and a slow one that operate on different timescales but share a memory module. This means that one of them has a very accurate look at the near past and the other one has a more coarse look but can look back more into the past in return. If these two work together correctly, decisions can be made that are both good locally at this point in time and and globally to maximize the probability of winning the whole game. This is really huge because this algorithm can perform long term planning, which is one of the key reasons why many difficult games and tasks remain unsolved. Well, as it seems now, not for long. An additional challenge is that the game score is not necessarily subject to maximization like in most games, but there is a mapping from the scores into an internal reward, which means that the algorithm has to be able to predict its own progress towards winning. And note that even though Quake 3 and capture the Flag is an excellent way to demonstrate the capabilities of this algorithm, this architecture can be generalized to other problems. I am going to give you a few more tidbits that I have found super interesting. But before if you are enjoying this episode and would like to pick up some cool perks like early access, deciding the topic of future episodes, or getting your name listed in the video description as a key supporter, why not support the show on Patreon? Don't forget to with this you can also help us make better videos in the future. You can find us at patreon.com twominutepapers and we also support Bitcoin and other cryptocurrencies. The addresses are available in the video description. And now onwards to the cool tidbits. A human plus agent team has been able to defeat an agent plus agent team 5% of the time, indicating that these AIs are able to coordinate and play together with anyone they are given. I get goosebumps from this. Love it. The reaction time and accuracy of the agents is better than that of humans, but not nearly perfect as many people would think. However, they outclass humans even if we artificially reduce their accuracy and reaction times. In another experiment, two agents were paired up against two professional game tester humans who could freely communicate and train against the Same agents for 12 hours to see if they can learn their patterns and force them to make mistakes. Even with this, humans had only won 25% of these games. Given the other numbers we have, it is very likely that this unfair advantage made no difference whatsoever. How about that? If there are any more questions, make sure to have a look at the paper that describes every possible tidbit you can possibly imagine. Thanks for watching and for your generous support and I'll see you next time.
