Short explainer video describing how meta-learning of reusable sub-policies (shared hierarchies) speeds up reinforcement learning and generalization across tasks. The host explains the inefficiency of learning from scratch, argues that sub-policies reduce search complexity and transfer between tasks, cites Neural Task Programming and an ants-in-mazes example, notes source code availability, and ends with a channel recommendation and CTA.
Chapters
Dear fellow scholars, this is two minute papers with Reinforcement learning is a technique where we have a virtual creature that tries to learn an optimal set of actions to maximize a reward in a changing environment. Playing video games, helicopter control, and even optimizing light transport simulations are among the more awesome example use cases for it. But if we train a reinforcement learner from scratch, we'll see that it typically starts out with a brute force search in the space of the simplest, lowest level actions. This not only leads to crazy behavior early on, but is also highly ineffective, requires way more experience than humans do, and the obtained knowledge cannot be reused for similar tasks. It can learn the game it was trained on, often even on a superhuman level, but if we need it to function in a new environment, all this previous knowledge has to be thrown away. And this algorithm is very much like how humans learn. It breaks down a big and complex task into sequences of smaller actions. These are called sub policies and can be shared between tasks. Learning to walk and crawl are excellent examples of that and will likely be reused for a variety of different problems and will lead to rapid learning on new unseen tasks, even if they differ significantly from the previously seen problems. Not only that, but the search space over sub policies can easily be a hundred or more times smaller than the original search space of all possible actions. Therefore, this kind of search is way more efficient than previous techniques. Of course, creating a good selection of sub policies is challenging because they have to be robust enough to be helpful on many possible tasks, but not too specific to one problem, otherwise they lose their utility. A few episodes ago we mentioned a related technique by the name Neural task programming, and it seems that this one is capable of generalization not only over different variations of the same task, but across different tasks as well. These ants were trained to traverse several different mazes one after another, and quickly realized that the basic movement directions should be retained. Creating more general learning algorithms is one of the holy grail problems of AI research, and this one seems to be a proper, proper step towards defeating it. We are not there yet, but it's hard not to be optimistic with this incredible rate of progress each year. Really excited to see how this area improves over the next few months. The source code of this project is also available. Oh, and before we go, make sure to check out the channel of Robert Miles who makes excellent videos about AI, and I recommend starting with one of his videos that you are objectively guaranteed to enjoy. If you wish to find out why, you'll see the link in the video description or just click the cat picture appearing here on the screen in a moment. If you indeed enjoyed it, make sure to subscribe to his channel. Thanks for watching and for your generous support, and I'll see you next time.
