Károly Zsolnai-Fehér breaks down a DeepMind paper on training AI agents to do what humans actually want. DeepMind proposes letting users give feedback during training to update the reward function, rather than trying to specify the goal perfectly upfront. The approach was tested on Atari games.
Dear fellow scholars, this is two minute Papers with this episode does not have the usual visual fireworks, but I really wanted to cover this paper because it tells a story that is, I think, very important for all of us to hear about. When creating a new AI to help us with a task, we have to somehow tell this AI what we consider to be a desirable solution. If everything goes well, it will find out the best way to accomplish it. This is easy when playing simpler video games because we can just tell the algorithm to maximize the score seen in the game. For instance, the more bricks we hit in Atari Breakout, the closer we get to finishing the level. However, in real life, we don't have anyone giving us a score to tell us how close we are to our objective. What's even worse, sometimes we have to make decisions that seem bad at the time, but will serve us well in the future. Trying to save money or studying for a few years longer are typical life decisions that pay off in the long run, but may seem undesirable at the time. The opposite is also true. Ideas that may sound right at the time may immediately backfire. When in a car chase, don't ask the car AI to unload all unnecessary weights to go faster. Or if you do, prepare to be promptly ejected from the car. So how can we possibly create an AI that somehow understands our intentions and acts in line with them? That's a challenging question and is often referred to as the agent alignment problem. It has to be aligned with our values. What can we do about this? Well, short of having a mind reading device, we can maybe control the behavior of the AI through its reward system. Scientists at DeepMind just published a paper on this topic, where they started their thought process from two assumptions, quoting the authors. For many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior. In short, it is easier to yell at the TV than to become an athlete. Sounds reasonable, right? Note that from complexity theory we know that this does not always hold, but it is indeed true for a large number of difficult problems. User intentions can be learned with high accuracy. In other words, given enough data that somehow relates to our intentions, the AI should be able to learn that. Leaning on these two assumptions, we can change the basic formulation of reinforcement learning in the following way. Normally we have an agent that chooses a set of actions in an environment to maximize a score. For instance, this could mean moving the pedal around to hit as many blocks as possible and finish the level. They extended this formulation in a way that the user can periodically provide feedback on how the score should be calculated now the AI will try to maximize this new score and we hope that this will be more in line with our intentions. Or in our car chase example, we could modify our reward to make sure we remain in the car and not get ejected. Perhaps the most remarkable property of this formulation is that it doesn't even require us to, for instance, play the game at all to demonstrate our intentions to the algorithm. The formulation follows our principles and not our actions. We can just sit in our favorite armchair, bend the AI to our will by changing the reward function every now and then and and let the AI do the grueling work. This is like yelling at the tv, except that it actually works. Loving the idea. If you have a look at the paper you will see a ton more details on how to do this efficiently and a case study with a few Atari games. Also, since this has a lot of implications pertaining to AI safety and how to create aligned agents, an increasingly important topic these days. Huge respect for DeepMind for investing more and more of their time and money in this area. Thanks for watching and for your generous support and I'll see you next time.
