Deep Learning From Human Preferences | Two Minute Papers #196

Dear fellow scholars, this is Two Minute Papers with In this new age of AI, there is no shortage of articles and discussion about AI safety. And of course, rightfully so. These new learning algorithms started solving problems that were previously thought to be impossible in quick succession only 10 years ago. If we told someone about half of the things that have been covered in the last few 2 Minute Papers episodes, we'd have been declared insane. We and of course, having such powerful algorithms, we have to make sure that they are used for good. This work is a collaboration between OpenAI and DeepMind security team and is about introducing more human control in reinforcement learning problems. The goal was to learn to perform a backflip through reinforcement learning. This is an algorithm that tries to perform a series of actions to maximize a score, kind of like playing computer games. For instance in Atari Breakout. If we break a lot of bricks, we get a high score so we know we did something well. If we see that happening, we keep doing what led to this result. If not, we go back to the drawing board and try something new. But this work is about no ordinary reinforcement learning algorithm because the score to be maximized comes from a human supervisor, and we are trying to teach a digital creature to perform a backflip. I particularly like the choice of the backflip here because we can tell when we see one, but the mathematical specification of this in terms of movement actions is rather challenging. This is a problem formulation in which humans can overlook and control the learning process, which is going to be an increasingly important aspect of learning algorithms in the future. The feedback option is very simple. We just specify whether this sequence of motions achieved our prescribed goal or not. Did it fall or did it perform the backflip successfully? After around 700 human feedbacks, the algorithm was able to learn the concept of a backflip, which is quite remarkable given that these binary yes no scores are extremely difficult to use for any sort of learning. In an earlier episode we illustrated a similar case with a careless teacher who refuses to give out points for each problem on a written exam and only announces whether we have failed or passed. This clearly makes a dreadful learning experience, and it is incredible that the algorithm is still able to learn using these. We provide feedback on less than 1% of the actions the algorithm makes, and it can still learn difficult concepts off of these extremely sparse and vague rewards. Low quality teaching leads to high quality learning. How about that? This is significantly more complex than what other techniques were able to learn with human feedback, and it works with other games too. A word about the collaboration itself. When a company hires a bunch of super smart scientists and spends a ton of money on research, it is understandable that they want to get an edge through these projects, which often means keeping the results for themselves. This leads to excessive secrecy and a lack of collaboration with other groups as everyone wants to keep their cards close to their chest. The fact that such collaborations can happen between these two AI research giants is a testament to how devoted they are to working together and sharing their findings with everyone free of charge for the greater good. Awesome. As the media is all up in arms about the demise of the human race, I feel that it is important to show the other side of the coin as well. We have top people working on AI safety right now. If you wish to help us tell these stories to more and more people, please consider supporting us on Patreon. Details are available in the video description or just click the letter P that appears on the screen in a moment. Thanks for watching and for your generous support and I'll see you next time. Sam.