DeepMind's Imagination-Augmented Agents let AI imagine future moves before acting, solving long-term planning problems that stumped earlier methods. The system simulates action sequences to pick plans with high expected reward, and solved a seven-box Sokoban puzzle that required irreversible decisions.
Dear fellow scholars, this is two minute papers with a bit more than two years ago, the DeepMind guys implemented an algorithm that could play Atari Breakout on a superhuman level by looking at the video feed that you see here. And the news immediately took the world by storm. This original paper is a bit more than 2 years old and has already been referenced in well over a thousand other research papers. That is one powerful paper. This algorithm was based on a combination of a neural network and reinforcement learning. The neural network was used to understand the video feed and reinforcement learning is there to come up with the appropriate actions. This is the part that plays the game. Reinforcement learning is very suitable for tasks where we are in a changing environment and we need to choose an appropriate action based on our surroundings to maximize some sort of score. This score can be, for instance, how far we've gotten in a labyrinth, or how many collisions we have avoided with a helicopter, or any sort of score that reflects how well we are currently doing. And this algorithm works similarly to how an animal learns new things. It observes the environment, tries different actions and sees if they worked well. If yes, it will keep doing that. If not, well, let's try something else. Pavlov's Dog with the Bell is an excellent example of that. There are many existing works in this area and it performs remarkably well for a number of problems and computer games, but only if the reward comes relatively quickly after the action. For instance, in Breakout, if we miss the ball, we lose a life immediately, but if we hit it, we'll almost immediately break some bricks and increase our score. This is more than suitable for a well built reinforcement learner algorithm. However, this earlier work didn't perform well on any games that required long term planning. If Pavlov gave his dog a treat for something that it did two days ago, the animal would have no clue as to which action led to this tasty reward. And this work subject is a game where we control this green character and our goal is to push the boxes onto the red dots. This game is particularly difficult, not only for algorithms, but even humans because of two important reasons. One, it requires long term planning, which as we know is a huge issue for reinforcement learning algorithms. Just because a box is next to a dot doesn't mean that it is the one that belongs there. This is a particularly nasty property of the game. And two, some mistakes we make are irreversible. For instance, pushing a box in a corner can make it impossible to complete the level. If we have an algorithm that tries a bunch of actions and sees if they stick well that's not going to work here. And it is now hopefully easy to see that this is an obscenely difficult problem and the DeepMind guys just came up with Imagination Augmented Agents as a solution for it. So what is behind this really cool name? The interesting part about this novel architecture is that it uses imagination, which is a routine, to cook up not only one action, but entire plans consisting of several steps, and finally, choose one that has the greatest expected reward over the long term. It takes information about the present and imagines possible futures and chooses the one with the most handsome reward. And as you can see, this is only the first paper on this new architecture and it can already solve a problem with seven boxes. This is just unreal. Absolutely amazing work. And please note that this is a fairly general algorithm that can be used for a number of different problems. This particular game was just one way of demonstrating the attractive properties of this new technique. The paper contains more results and is a great read. Make sure to have a look. Also, if you've enjoyed this episode, Please consider supporting 2 minute papers on Patreon. Details are available in the video description. Have a look. Thanks for watching and for your generous support and I'll see you next time.
