Dr. Zsolnai shows how an adaptive curriculum trains AI agents to climb difficult terrain. A virtual human, a Cassie biped robot, and a sphere-shaped creature all learn to navigate stepping stones by starting easy and getting harder tasks. After 12 to 24 hours of training, they handle steep drops, tilted ground, and rocky streams.
Questions this video answers
February 19, 2021
An OpenAI paper on self-play where virtual agents learned stable stances in a sumo-like task (2017 OpenAI self-play/sumo work)
Virtual human, Cassie (bipedal robot), and a sphere called 'Monster'
About 12 to 24 hours
Quadro RTX 6000, RTX 8000, and V100; also notes a 48 GB RTX 8000 instance
Dear fellow scholars, this is two minute.
Papers with Dr. Zsolnai. In 2017, scientists at OpenAI published a paper where virtual humans learned to tackle each other in a sumo competition of sorts and found out how to rock a stable stance to block others from tackling them. This was a super interesting work because it involved self play, or in other words, copies of the same AI were playing against each other and the question was, how do we pair them with each other to maximize their learning? They found something really remarkable when they asked the algorithm to defeat an older version of itself. If it can reliably pull that off, it will lead to a rapid and predictable learning process. This kind of curriculum driven learning can supercharge many different kinds of AIs. For instance, this robot from a later paper is essentially blind, as it only.
Has proprioceptive sensors, which means that the.
Only thing that the robot senses is its own internal state. And that's it. No cameras, no depth sensors, no lidar, nothing. And at first it behaves as we would expect it. Look, when we start out, the agent is very clumsy and can barely walk through a simple terrain. But as time passes, it grows to be a little more confident, and with that, the terrain also becomes more difficult over time in order to maximize learning. So how potent is this kind of curriculum in teaching the AI? Well, it learned a great deal in the simulation. And as scientists deployed it into the real world, just look at how well it traversed through this rocky mountain stream. And not even this nightmarish snowy descent gave it too much trouble. This new technique proposes a similar curriculum based approach where we would teach all kinds of virtual life forms to navigate on stepping stones. The examples include a virtual human, a bipedal robot called Cassie, and this sphere with toothpick legs too. The authors call it Monster. So you know what monster it is. The fundamental question here how do we organize the stepping stones in this virtual environment to deliver the best teaching to this AI? We can freely choose the heights and orientations of the upcoming steps. And of course, it is easier said than done. If the curriculum is too easy, no meaningful learning will take place. And if it gets too difficult too quickly, well then in the better case this happens. And in the worst case, Whoops. This work proposes an adaptive curriculum that constantly measures how these agents perform and creates challenges that progressively get harder, but in a way that they can be solved by the agents. It can even deal with cases where the AI already knows how to climb up and down and even deal with longer steps. But that does not mean that we are done because if we don't build these spires right this happens. But after learning 12 to 24 hours with this adaptive curriculum learning method, they become able to even run deal with huge step height variations, high step tilt variations and let's see if they can pass the hardest exam. Look at this mess. My goodness. Lots of variation in every parameter and yes it works. And the key point is that the system is general enough that it can teach different body types to do the same. If there is one thing that you take home from this video, it it shouldn't be that it takes from 12 to 24 hours, it should be that the system is general. Normally if we have a no body type we need to write a no control algorithm. But in this case, whatever the body type is, we can use the same algorithm to teach it. Absolutely amazing. What a time to be alive. However, I know what you're thinking. Why teach them to navigate Just stepping stones? This is such a narrow application of locomotion, so why this task? Great question and the answer is that the generality of this technique we just talked about also means that the stepping stone navigation truly was just a stepping stone. And here it is. We can deploy these agents to a continuous terrain and expect them to lean on their stepping stone chops to navigate well here too. Another great triumph for curriculum based AI training environments. So what do you think? What would you use this technique for? Let me know in the comments. Or if you wish to discuss similar topics with other fellow scholars in a warm and welcoming environment, make sure to join our Discord channel. The link is available in the video.
Description this episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus they are the only cloud service with 48gb RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make.
Better videos for you. Thanks for watching and for your generous support and I'll see you next time.
