Can We Teach a Robot Hand To Keep Learning?

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. In 2019, researchers at OpenAI came up with an amazing learning algorithm that they deployed on a robot hand that was able to dexterously manipulate a Rubik's cube even when it was severely hamstrung. A good game plan to perform such a thing is to first solve the problem in a computer simulation where we can learn and iterate quickly and then transfer everything the agent learned there to the real world and hope that it obtained general knowledge that indeed can be applied to real tasks. Papers like these are some of my favorites. If you are one of our core fellow scholars, you may remember that we talked about walking robots about 200 episodes ago. In this amazing paper, we witnessed a robot not only learning to walk, but it could also adjust its behavior and keep walking even if one or multiple legs lose power or get damaged. In this previous work, the key idea was to allow the robot to learn tasks such as walking not only in one optimal way, but to explore and build a map of many alternative motions relying on different body parts. Both of these papers teach us that working in the real world often shows us new unexpected challenges to overcome. And this new paper offers a technique to adapt a robot arm to these challenges. After it has been deployed into the real world, it is supposed to be able to pick up objects, which sounds somewhat simple these days until we realize that new previously unseen objects may appear in the bin with different shapes or material models. For example, reflective and refractive objects are particularly perilous because they often show us more about their surroundings than about themselves. Lighting conditions may also change after deployment. The gripper's length or shape may change, and many, many other issues are likely to arise. Let's have a look at the lighting conditions part. Why would that be such an issue? The objects are the same, the scene looks nearly the same. So why is this a challenge? Well, if the lighting changes, the reflections change significantly, and since the robot arm sees its reflection and thinks that it is a different object, it just keeps trying to grasp it. After some fine tuning, this method was able to increase the otherwise not too pleasant 32% success rate to 63% much, much better. Also, extending the gripper used to be somewhat of a problem, but as you see here, with this technique, it is barely an issue anymore. Also, if we have a somewhat intelligent system and we move the position of the gripper around, nothing really changes, so we would expect it to perform well. Does it? Well, let's have a look. Unfortunately, it just seems to be rotating around without too many meaningful actions. And now hold on to your papers because after using this continual learning scheme, yes, it improved substantially and makes very few mistakes and can even pick up these tiny objects that are very challenging to grasp with this clumsy hand. This fine tuning step typically takes an additional hour or or at most a few hours of extra training and can be used to help these AIs learn continuously after they are deployed in the real world, thereby updating and improving themselves. It is hard to define what exactly intelligence is, but an important component of it is being able to reuse knowledge and adapt to new unseen situations. This is exactly what this paper helps with. Absolute witchcraft. What a time to be alive. This episode has been supported by Linode. Linode is the world's largest independent cloud computing provider. Unlike entry level hosting services, Linode gives you full backend access to your server, which is a step up to powerful, fast, fully configurable cloud computing. Linode also has one click apps that streamline your ability to Deploy websites, personal VPNs, email, game servers and more. If you need something as small as a personal online portfolio, Linode has your back. And if you need to manage tons of clients websites and reliably serve them to millions of visitors, Linode can do that too. What's more, they offer affordable GPU instances featuring the Quadro RTX 6000, which is tailor made for AI, scientific computing and computer graphics projects. If only I had access to a tool like this while I was working on my last few papers. To receive $20 in credit on your new Linode account, visit linode.com papers or click the link in the description and give it a try today. Our thanks to Linode for supporting the series and helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.