DeepMind's Agent 57 beats humans at all 57 Atari games by using a meta-controller that balances short-term skill with long-term exploration. Earlier AI agents struggled with games like Montezuma's Revenge that require planning far ahead, but Agent 57 solves this by learning when to focus on immediate tactics versus searching for new strategies.
Dear fellow scholars, this is 2 Minute Papers with Dr. Zsolnai. Between 2013 and 2015, DeepMind worked on an incredible learning algorithm by the name Deep Reinforcement Learning. This technique looked at the pixels of the game, was given a controller and played much like a human would. With the exception that it learned to play some Atari games on a superhuman level. I have tried to train it a few years ago and would like to invite you for a marvelous journey to see what happened when it starts learning to play an old game, Atari Breakout. At first the algorithm loses all of its lives without any signs of intelligent action. If we wait a bit, it becomes better at playing the game, roughly matching the skill level of an adept player. But here's the catch. If we wait for longer, we get something absolutely spectacular. Over time, it learns to play like a pro and finds out that the best way to win the game is digging a tunnel through the bricks and hit them from behind. This technique is a combination of a neural network that processes the visual data that we see on the screen and a reinforcement learner that comes up with the gameplay related decisions. This is an amazing algorithm, a true breakthrough in AI research. However, it had its own issues. For instance, it did not do well on Montezuma's Revenge or Pitfall because these games require more long term planning. Believe it or not, the solution in a follow up work was to infuse these agents with a very human like property curiosity. That agent was able to do much, much better at these games and then got addicted to the tv. But that's a different story. Note that this has been remedied since and believe it or not, as impossible as it may sound, all of this has been improved significantly. This new work is called Agent 57 and it plays better than humans on all 57 Atari games. Absolute insanity. Let's have a look at it in action and then in a moment I'll try to explain how it does what it does. You see Agent 57 doing really well at the Solaris game here. This space battle game is one of the most impressive games on the Atari as it contains 16 quadrants, 48 sectors, space battles, warp mechanics, pirate ships, fuel management, and much more. You name it. This game is not only quite complex but but it also is a credit assignment nightmare for an AI to play. This credit assignment problem means that it can happen that we choose an action and we only win or lose hundreds of actions later, leaving us with no idea as to which of our actions led to this win or loss. Thus making it difficult to learn from our Actions. This Solaris game is a credit assignment nightmare. Let me try to bring this point to life by talking about school. In school, when we take an exam, we hand it in and the teacher gives us feedback for every single one of our solutions and tells us whether we were correct or not. We know exactly where we did well and what we need to practice to do better next time. Clear, simple, easy. Solaris, on the other hand, not so much. If this were a school project, the Solaris game would be a brutal, merciless teacher. Would you like to know your grade? No grades, but he tells you that you failed. Well, that's weird. Okay, where did we fail? He won't say. What should we do better next time to improve? You'll figure it out, bucko. Also, we wrote this exam 10 weeks ago. Why do we only get to know about the results now? No answer. I think in this case, we can conclude that this would be a challenging learning environment, even for a motivated human. So just imagine how hard it is for an AI. Hopefully this puts into perspective how incredible it is that Agent 57 performs well on this game. It truly looks like science fiction. To understand what Agent 57 adds to this, it was given something called a meta controller that that can decide when to prioritize short and long term planning. On the short term, we typically have mechanical challenges like avoiding a skull in Montezuma's Revenge or dodging the shots of an enemy ship in Solaris. The long term part is also necessary to explore new parts of the game and have a good strategic plan to eventually win the game. This is great because because this new technique can now deal with the brutal and merciless teacher who we just introduced. Alternatively, this agent can be thought of as someone who has a motivation to explore the game and do well at mechanical tasks at the same time, and can also prioritize these tasks with this. For the first time, scientists at DeepMind found a learning algorithm that exceeds human performance on all 57 Atari games. And please do not forget about the fact that DeepMind tries to solve general intelligence and then use general intelligence to solve everything else. This is their holy grail. In other words, they are seeking an algorithm that can learn by itself and achieve human like performance in a variety of tasks. There is still plenty to do, but we are now one step closer to that. If you learn only one thing from this video, let it be the fact that There are not 57 different methods, but one general algorithm that plays 57 games better than humans. What a time to be alive. I would like to show you a short message from a few days ago that melted my heart. This I got from Nathan who has been inspired by these incredible works and he decided to turn his life around and and go back to study more. I love my job and reading messages like this is one of the absolute best parts of it. Congratulations Nathan and note that you can take this Inspiration and greatness can materialize in every aspect of life, not only in computer graphics or machine learning research. Good luck. If you're a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lamda GPU Cloud I've talked about Lambda's GPU workstations in other videos and am happy to tell you that they are offering GPU cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers because the Lambda GPU Cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs. Com papers and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
