DeepMind trained a robot to insert a USB stick with 80% accuracy after 8 hours of training, using a reward-sketching method that also lets it handle deformable objects and adapt to new items. The robot performs the task faster than a human demonstrator.
About 80%.
Approximately 8 hours of training.
Dr. Károly Zsolnai-Fehér (Two Minute Papers).
OpenAI (prior) and DeepMind (current).
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Two years ago, in 2019, scientists at OpenAI published an outstanding paper that showcased a robot arm that could dexterously manipulate a Rubik's cube. And how good were the results? Well, nothing short of spectacular. It could not only manipulate and solve the cube, but but they could even hamstring the hand in many different ways and it would still be able to do well. And I am telling you, scientists at OpenAI got very creative in tormenting this little hand. They added a rubber glove, tied multiple fingers together, threw a blanket on it and pushed it around with a plush giraffe and a pen. It still worked, but probably had nightmares about that day in the lab. Robot nightmares, if you will. And now let's see what is going on over DeepMind's side and have a look at this work on manipulating objects. In this example, a human starts operating this contraption and after that step is done, we leave it alone. And our first question, how do we tell this robot arm what it should do? It doesn't yet know what the task is, but we can tell it by this reward sketching step. Essentially, this works like a video game. Not in the classical setting, but the other way around. Here we are not playing the video game, but we are the video game. And the robot is the character that plays the video game in which we can provide it feedback and tell it to when it is doing well or not. Then from these rewards, it can learn what the task is. So all that sounds great, but what do we get for all this work? Four things. One, we can instruct it to learn to lift deformables. This includes a piece of cloth, a rope, a soft ball, and more. Two, much like OpenAI's robot arm, it is robust against perturbations. In other words, it can recover from our evil machinations. Look, we can get creative here too. For instance, we can reorganize the objects on the table, nudge an already grabbed object out of its gripper, or simply just undo the stacks that are already done. This diligent little AI is not phased. It just keeps on trying and trying and eventually it succeeds. A great life lesson right there. And three, it generalizes to new object types well and does not get confused by different object colors and geometries. And now hold on to your papers for number four, because here comes one of the most frustrating tasks for any intelligent being. Something that not many humans can perform. Inserting a USB key correctly on the first try. Can it do that? Well, does this count as a first try? I don't know for sure, but dear fellow scholars, the singularity is officially getting very, very close. Especially given that we can even move the machine around a little and it will still find the correct port. If only the first machine could pronounce my name correctly, then we would conclude that we have reached the singularity. But wait, we noted that first a human starts controlling the arm. How does the fully trained AI compare to this human's work? And here comes the best part. It learned to perform these tasks faster. Look. By the six second mark the human started grabbing the green block. But by this time the AI is already mid air with it and by 9 seconds it is done while the human is still at work. Excellent. And we get an 80% success rate on this task with only 8 hours of training, that is within a single working day. One of the key ideas here, and the part that I liked best, is that we can essentially reprogram the AI with the reward sketching step. And I wonder what else this could be used for if we did that. Do you have some ideas? Let me know in the comments below. This episode has been supported by Weights and Biases. In this post they show you how to use their tool to draw bounding boxes for object detection and even more importantly, how to debug them. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
