OpenAI Performs Surgery On A Neural Network to Play DOTA 2

Dear fellow scholars, this is two minute papers with finally the full research paper has appeared on OpenAI 5, which is an AI that plays Dota 2, a multiplayer online battle arena game with a huge cult following, and you may not expect this. It is not only as good as some of the best players in the world, but it also describes a surgery technique that sounds quite unexpected and I promise to tell you what it is later during this video. This game is a nightmare for any AI to play because of three main reasons. One, it requires long term strategic planning where it is possible that we make one bad decision, then a thousand good ones and we still lose the game in the end. Finding out which decision led to this loss is immensely difficult, often even for humans. Two, we have imperfect information, meaning that we can only see what our units and buildings can see. And three, even though these learning agents don't look at the pixels of the game, but they see the world as a big bunch of numbers, there's just too much information to look at and too many decisions to make compared to chess or Go or almost anything else. Despite these difficulties, in 2017 OpenAI showed us an initial version of their agent that was able to play one versus one games with only one hero and was able to reliably beat Dandy, but a world champion player. That was quite an achievement. However, of course this was meant to be a stepping stone towards something much bigger, that is playing the real Dota 2. And just two years later, a newer version named OpenAI 5 has appeared, defeated the Dota 2 world champions, and beat 99.4% of human players during an online event that ran for multiple days. Many voices said that this would never happen, so two years to pull this off after the first version I think was an absolute miracle. Bravo. Now note that even this version has two key limitations. One, in a normal game, we can choose from a pool of 117 heroes where this system supports 17 of them, and two, items that allow the player to control multiple characters at once have been disabled. If I remember correctly from a previous post of theirs, invisibility effects are also neglected because the algorithm is not looking at pixels. It would either always have this information shown as a bunch of numbers, or never. Neither of these would be good design decisions, so thus invisibility is not part of this technique. Fortunately, the paper is now available, so I was really excited to look under the hood for some more details. So first, as I promised, what is this surgery thing about? You see, the training of the neural network part of this algorithm took no less than 10 months. Now, just imagine forgetting to feed an important piece of information into the system or finding a bug while training is underway. In cases like this, normally we would have to abort the training and start again. If we have a new idea as to how to improve the system again, we have to abort the training and start again. If a new version of Dota 2 comes out with some changes, you guessed right, we start again. This would be okay if the training took from the order of minutes to hours, but we are talking 10 months here. This is clearly not practical. So what if there would be a technique that would be able to apply all of these changes to a training process that is already underway? Well, this is what the surgery technique is about. Here with the blue curve, you see the agent's skill rating improving over time. And the red lines with the black triangles show us the dates for the surgeries. The authors note that over the 10 month training process, they have performed approximately one surgery per two weeks. It seems that getting a doctorate in machine learning research is getting a whole new meaning. Some of them indeed made an immediate difference, while others seemingly not so much. So how do we assess how potent these surgeries were? Did they give the agent superpowers? Well, have a look at the rerun part here, which is the final Frankenstein's monster agent, containing the result of all the surgeries retrained from scratch. And just look at how quickly it is trained. And not only that, but it shoots even higher than the original agent. Absolute madness. Apparently, OpenAI is employing some proper surgeons over there at their lab. I love it. Interestingly, this is not the only time I've seen the word surgery used in the computer sciences outside of medicine. A legendary mathematician named Grigori Perelman, who proved the Poincare conjecture, also performed a mathematical technique that he called surgery. What's more, we even talked about simulating weightlifting and how a simulated AI agent will walk after getting hamstrung. And you guessed it right, undergoing surgery to fix it. What a time to be alive. And again, an important lesson is that in this project, OpenAI is not spending so much money and resources just to play video games. Dota 2 is a wonderful testbed to see how their AI compares to humans at complex tasks that involve strategy and teamwork. However, the ultimate goal is to reuse parts of this system for other complex problems outside of video games. For instance, the algorithm that you've seen here today can also do this. This episode has been supported by Linode. Linode is the world's largest independent cloud computing provider. Unlike entry level hosting services, Linode gives you full backend access to your server, which is your step up to powerful, fast, fully configurable cloud computing. Linode also has one click apps that streamline your ability to deploy websites, personal VPNs, game servers and more. If you need something as small as a personal online portfolio, Linode has your back. And if you need to manage tons of clients websites and reliably serve them to millions of visitors, Linode can do that too. What's more, they offer affordable GPU instances featuring the Quadro RTX 6000 which is tailor made for AI, scientific computing and computer graphics projects. If only I had access to a tool like this while I was working on my PhD studies. To receive $20 credit in your new Linode account, visit linode.com papers or just click the link in the video description and give it a try today. Our thanks to Linode for supporting the series and helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.