Dr. Zsolnai covers a new physics simulator that teaches robots to grasp and manipulate objects by training them in a detailed digital world that closely matches reality. The system combines rigid and soft bodies with optical simulation, shrinking the gap between what robots learn in simulation and what actually works when they try it in the real world.
It is crazy that no one is talking about this. This work was done as a collaboration between several research labs across the world and is about creating physics simulations on a computer to teach AIs about the world. And what are the simulations about? Well, to help robots noodle around with things. Yes, I hear you saying Caroj. What does that mean? Let me try to explain. When we try to teach a robot to navigate safely in the real world, simply first we don't just drop them on the street and let them learn the basics there. Not at all. What we do instead is that we put them in a simulated environment where we can give it crazy hard tasks. And when it has demonstrated that it has learned everything and is safe, then we can consider testing them in reality. This is called Sim2Real. Going from simulation to reality, self driving cars and robots are doing better and better in this regard. We have these little computer games where they can train. But for everyone else there are two huge problems. Problem number one, for robots that are not driving, but moving objects around, they keep on touching things. Unfortunately they don't have a really good game to play yet. But wait, is grasping that hard? Well, for us it is not hard at all. By the time we are small children, this is more or less child's play for us. But for robots, yes, it is incredibly hard. Look, if it grasps too lightly, this happens. And if it grasps too hard, the results are quite predictable. It has to do everything with just the right amount of force. And that needs tons and tons of learning and tons of training data. So for these robots that have tactile sensors, this fantastic new paper contains a video game for virtual robots to play in. But it gets better it. In this game they are meant to touch things and based on how they feel, follow their surfaces, open a box, move an object and so on. But unfortunately we are still not ready to solve this grasping task because we have one more huge problem. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. So what is problem number two? It is the sim to real gap. A simulation is great. However, reality is not like a video game. You can try to simulate everything into the tiniest detail, but when entering the real world, plans go wrong all the time. So does it make sense to have a video game that does not teach reality, but it teaches useful knowledge in a made up world? Of course it doesn't. So what can we do with that? Oh my, look at that. This is a differentiable system, which means that we can go from simulation to reality. And when we see that there is a gap between. We can account for the differences and automatically reprogram it to match reality much more closely. In short, we can close the gap between simulation and reality. That is absolutely incredible and is one of the key elements to make useful robots to fold our laundry in the future. So can it finally do the grasping problem? We will see in just a moment. Now, there are many previous systems that can do parts of what this is doing. Rigid bodies, soft bodies, differentiability, optical simulations, they can do parts of it. But this new system seems to be the only one that has it all. And oh, look at that. It uses Taichi as a backend. Yes, fellow scholars, this is something that we talked about approximately 600 videos ago. Yes, I think this might be the only corner of the Internet where you see the history of these research works unfold right before your eyes. Amazing. I love my work. Now grasping, you remember that too soft was not great. Grasping too hard, also not great. But after learning in this simulated environment, let's see. Yup, that is just the right amount of force. This concept really works. So cool. So I would like to send a huge thank you for these brilliant scientists for giving this away for all of us for free. And of course a big thank you to this cat too, who surely is the mastermind behind all this. It is probably thinking, finally, I am inventing robots that play patty cake with us. What a time to be alive. I mean, when cats do it among themselves, it always ends up in a brutal fight. Look at that. And I hear so few people talking about this. It's crazy. And this is why 2 minute papers exist, not to create videos that are 2 minutes long. I have clearly failed mightily at that. It exists to talk about amazing research papers and amazing human achievements that very few people if anyone is talking about. So subscribe and hit the bell icon if you wish to hear more. And what would you fellow scholars use this for? Let me know in the comments below. We need new tools for the era of LLMs and weights and biases. Now has weave a lightweight toolkit to confidently iterate on LLM applications. Use traces to debug how data flows through each step of your app, and use evaluations to measure your progress. It is the best. Try it out now at WNB Me papersllm or click the link in the description below.
