DeepMind and Blizzard released an open-source StarCraft II learning environment for AI research. The system handles partial observability, large-scale unit control, and long-term planning through deep reinforcement learning. Agents trained on minigames reached novice human level, with the team publishing game recordings and full source code.
Questions this video answers
August 23, 2017.
A Python-based interface.
The "Blizzard score" — a weighted sum of current resources, upgrades, units, and buildings.
Chapters
Dear fellow scholars, this is two minute papers with this topic has been perhaps the most highly anticipated by you fellow scholars. And I am extremely excited to show you the first joint paper between DeepMind and Blizzard on creating an AI program to play Starcraft II. Hell yeah. And fortunately we have a paper where every detail is meticulously described, so there's much less room for misunderstandings. And before we start, note that this is a preliminary work, so please don't expect superhuman performance. However difficult you thought this problem was, you'll see in a minute that it's way more complex than most people would think. But before we start, what is Starcraft 2? It is a highly technical strategy game which will be a huge challenge to write a formidable AI for because of three one, we have imperfect information with a partially observed map. If we wish to see what the opponent is up to, we have to devote resources to scouting, which may or may not be successful depending on the vigilance of the other player. Two, we need to select and control hundreds of units under heavy time pressure. One wrong decision and we can quickly lose most of our units and become unable to recover from it. And three, perhaps the most important part, long term strategies need to be developed where a poor decision in the early game can lead to a crushing defeat several thousands of actions later. These cases are especially difficult to identify and learn. However, we ran a bit too far ahead to the gameplay part. What needs to be emphasized is that there is a step number one before that and that step number one is making sure that the AI is able to communicate and interact with the game, which requires a significant engineering effort. In this paper a python based interface is described to make this happen. It is great to have companies like DeepMind and OpenAI who are devoted to lay down the foundations for such an interface. Interface which is a herculean task. This work would likely had never seen the light of day if AI research would only take place in academia. Huge respect and much thanks for the DeepMind guys for making this happen. To play the game, deep reinforcement learning is used which you heard about earlier in the series. This is a powerful learning algorithm where a neural network is used to process the video input and is combined with a reinforcement learner. With reinforcement learning we are observing the environment around us and choose the next action to maximize the score or reward. However, defining score was very easy in Atari Breakout because we knew that if the number of our lives drops to zero, we lost and if we break a lot of bricks, our score improves. Simple. Not so much in Starcraft 2 because how do we know exactly if we are winning? What is the score we are trying to maximize? In this work there are two definitions for score. One that we get to know at the very end that describes whether we won, had a tie or lost. This is the score that ultimately matters. However, this information is not available throughout the game to drive the reinforcement learner. So there is an intermediate score that is referred to as Blizzard score in the paper, which involves a weighted sum of current resources and upgrades as well as our units and buildings. This sounds good for a first approximation since it is monotonically increasing if we win battles and manage our resources well and decreases when we are losing. However, there are many matches where the player with the more resources does not have enough time to spend it and ultimately loses a deciding encounter. It remains to be seen whether this is exactly what we need to maximize to beat a formidable human player. There are also non trivial engineering decisions on how to process the video stream. The current system uses a set of feature layers which encode relevant information for the AI, such as terrain height, the camera location, hit points for the units on the screen and much, much more. There is a huge amount of information that the convolutional neural network has to make sense of and I think it is now easy to see that starting out with throwing the AI in the deep water and expecting it to perform well on a full 1 vs 1 match at this point is a forlorn effort. The paper describes a set of minigames where the algorithm can learn different aspects of the game in isolation, such as picking up mineral shards scattered around the map, defeating enemy units in small skirmishes, building units or harvesting resources. In these minigames the AI has reached the level of a novice human player, which is quite amazing given the magnitude and the complexity of the problem. The authors also encourage the community to create more mini games for the AI to train on. I really love the openness and the community effort aspects of this work and we've only just scratched the surface. There is so much more in the paper with a lot more non trivial design decisions and the database with tens of thousands of recorded games and the best part is that the source code for this environment is available right now. For the fellow tinkerers out there, I've put a link to this in the video description. This is going to be one heck of a challenge for even the brightest AI researchers of our time. I can't wait to get my hands on the code and also I am very excited to read some follow up papers on this. I expect there will be many of those in the following months. In the meantime, as we know, OpenAI is also working on Dota with remarkable results, and there's lots of Discussion Whether a Dota 5 versus 5 or a Starcraft 2 one versus one game is more complex for the AI to learn. If you have an opinion on this, make sure to leave a comment below this video, which is more complex. Why? This also signals that there's going to be tons of fun to be had with AI and video games this year. Stay tuned. Thanks for watching and for your generous support, and I'll see you next time. Sam.
