OpenAI's Safety Gym teaches AI agents to learn tasks while avoiding dangerous behaviors. It uses constrained reinforcement learning to penalize safety violations like collisions, with applications from robot arms to self-driving cars and medical systems.
Dec 20, 2019
Linode
Papers20
Dear fellow scholars, this is two minute papers with Reinforcement learning is a technique in the field of machine learning to learn how to navigate in a labyrinth, play a video game, or to teach a digital creature to walk. Usually we are interested in a series of actions that are in some sense optimal in a given environment. Despite the fact that many enormous tomes exist to discuss the mathematical details, the intuition behind the algorithm itself is remarkably simple. Choose an action and if you get rewarded for it, try to find out which series of actions led to this and keep doing it. If the rewards are not coming, try something else. The reward can be, for instance, our score in a computer game, or how far our digital creature could walk. Approximately 300 episodes ago, OpenAI published one of their first major works by the name Jim, where anyone could submit their solutions and compete against each other on the same games. It was like Disney World for reinforcement learning researchers. A moment ago I noted that in reinforcement learning, if the rewards are not coming, we have to try something else. Hmm, is that so? Because there are cases where trying crazy new actions is downright dangerous. For instance, imagine that during the training of this robot arm, initially it would try random actions and start flailing about where it may damage itself, some other equipment, or even worse, humans may come to harm. Here you see an amusing example of DeepMind's reinforcement learning agent from 2017 that liked to engage in similar flailing activities. So what could be a possible solution for this? Well, have a look at this new work from OpenAI by the name Safety Gym. In this paper they introduce what they call the constrained reinforcement learning formulation in which these agents can be discouraged from performing actions that are deemed potentially dangerous in an environment. You can see an example here where the AI has to navigate through these environments and achieve a task such as reaching the green goal signs, push buttons, or move a box around to a prescribed position. The constraint part comes in whenever some sort of safety violation happens which are in this environment, collisions with the boxes or blue regions, all of these events are highlighted with this red sphere and a good learning algorithm should be instructed to try to avoid these. The goal of this project is that in the future for reinforcement learning algorithms, not only the efficiency but the safety scores should also be measured. This way, a self driving AI would be incentivized to not only drive recklessly to the finish line, but but respect our safety standards along the journey as well. While noting that clearly self driving cars may be achieved with other kinds of algorithms, many of which have been in the works for years, there are many additional applications for this work. For instance, the paper discusses the case of incentivizing recommender systems to not show psychologically harmful content to its users or to make sure that a medical question answering system does not mislead us with false information. This episode has been supported by Linode. Linode is the world's largest independent cloud computing provider. They offer you virtual servers that make it easy and affordable to host your own app, site, project or anything else in the cloud. Whether you are a Linux expert or just starting to tinker with your own code, Linode will be useful for you. A few episodes ago we played with an implementation of OpenAI's GPT2, where our excited viewers accidentally overloaded the system with Linode's load balancing technology and instances ranging from shared nanodes all the way up to dedicated GPUs. You don't have to worry about your project being overloaded. To get $20 of free credit, make sure to head over to linode.com papers and sign up today using the promo code Papers20 Our thanks to Linode for supporting the series and helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
