In this Two Minute Papers episode, Dr. Károly Zsolnai‑Fehér showcases "Sonic," a new teleoperated robot controller developed at NVIDIA. The video walks through how Sonic translates multimodal human inputs—video, voice, and even music—into whole‑body robot motions using a lightweight neural pipeline (encoder, quantizer, decoder) trained on 100 million frames of raw human motion without any hand‑crafted action labels. The model, only 42 million parameters, can run on a smartphone after an intensive 3‑day, 128‑GPU training run. A key technical contribution is the "root‑trajectory spring model," which damps rapid motions to keep the robot safe and stable. The narrator highlights potential applications such as search‑and‑rescue, hazardous‑environment exploration, and future planetary missions, emphasizing that the code and models are released openly for free. He also draws a philosophical parallel between the model’s token abstraction and how we should distill advice in life. The video ends with a sponsor plug for Lambda GPU cloud.
Chapters
Let's see what is going on here. This is me around 9 AM. A bit wobbly, steps are unsure. Yup, that checks out. Now then, give me my fake badge. Thank you, sir! Hehe, no one noticed. Now, let's proceed to the next step of my mastermind plan. Yes, let's eat all of their food! Wait! They noticed! Okay, proceed to the next step. What was that? Oh yes, run! Now, jokes aside, look at that. Oh yes, please mow my lawn! That is excellent! Now, rake the leaves. Perfect! Hey, don't slack off, that's my job! Okay, so what is going on here? Well, Let's start with the good news. This is a new teleoperated robot controller and more. They call it Sonic. Good name. Now, the work here is not the robot, but the software controlling the robot. At least in this footage. Watch until the end, and you might get surprised. So, teleoperation means there is a human performing these movements, and the robot is able to understand these motions and then translate them to a bunch of joint positions in 3D space. Whew! That's kind of insane if you think about it. But it will just get better and better as we continue the video. So, before you ask, yes, it can do kung fu, provided that you can do kung fu. It understands whole body movement, so you can get it to crawl into some space you don't want to go to. And that is super useful. People are already using robots for that. Why? Well, chiefly for exploring underexplored and dangerous areas. This means tons of useful applications, for instance, a variant of this could help save humans stuck under rubble. Or perhaps later, even explore other planets without putting humans at risk. But that is still nothing. Because this is a multimodal system. Meaning that the input can be almost anything. So, you say that I don't have to pretend to mow the lawn to actually get it to mow the lawn, because where is the fun in that? Well, you just tell it to do that. Can you? Well, currently, for simpler tasks like moving around and behaving like a monkey, Yes, you can! Absolutely incredible! And I love how expressive it is. You can ask it to walk happily, stealthily, or like an injured person. And, you know, just the fact that it is stable and does not fall is remarkable. Previously, even in simple characters, in simulated words, you needed thousands and thousands of tries to teach them to just be able to walk without falling. And now this. This is a huge leap forward. Wow! But it gets better. We said multimodal. Yup, that means the input can also be music. About that. I'll show you the dancing, but not the music because of YouTube reasons. But I'll put a link in the description where you can check it out. And we haven't even talked about the most insane part of the whole thing. Now! Hehehe. Hold on to your papers, Fellow Scholars, because this runs with about 42 million parameters. That is a neural network so simple, it can run so easily on your phone, it barely notices. It may even run on your toaster these days. That size is absolutely nothing. This is an incredible achievement. Okay, but how? How is that even possible? Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Well, first, it looked at 100 million frames of human motion to understand what we do and how we do it. The incredible thing is that this system does not require human-made action labels, so we don't have to explain our movements. It just watches the raw motions and figures out how to transition between tasks without any unnatural pauses. So then, your multimodal input goes in, a video of you, your voice, music, a motion generator turns these into human motion, and the human encoder processes it into a latent space, and then, a quantizer converts it to to universal tokens. Once again, universal tokens. That is the key, you'll see a bit later. Then, the decoder translates these tokens into motor commands. But— phew— there is a big problem. Learning to convert one to the other is super hard. First of all, robots do not work like humans, that is one of the fundamental challenges. So if the user commands you to turn around, it should be turning around. Okay, sure. But how fast exactly? You don't want to turn 180 degrees too quickly, because you would fall apart. To solve this, in their research paper, they propose what they call a root trajectory spring model. This looks for quick motions and dampens them. So the robot does not get injured. Yes, robots can get injured too, which is kind of hilarious. Now, there is an exponential term there as a function of time. What is that? That is a physical brake. As time increases, this term rapidly shrinks to zero, which forces the whole mathematical expression to decay smoothly. This serves two goals: 1. The robot does not injure itself. That's good. 2. It will settle at the target position without oscillating back and forth forever. Nice. Now, do the dampening too much, and— of course, you'll get a little slug that cannot get anything done. So it's really tough to do well. Now, all this took 128 GPUs and 3 days to train. That is expensive. But here's the key. After the training is done, the final product is so lightweight, we don't need this kind of hardware to run it at all. In fact, all of these models showcased in these videos will be given to all of us for free, forever. And they run on your phone, easy peasy. That is incredible. Open research for the benefit of humanity. Love it. Thank you so much! This project is led by Professors Zhu and Jim Fan, who I love dearly. Jim started the humanoid robots lab at NVIDIA just 2 years ago, and they are raining research papers on us, breakthrough after breakthrough. Haha. Insanity. And to compress all this human movement knowledge, down into a tiny little AI controller that can be used by any of us is simply a stunning achievement. It turns out training a good AI requires coding thinking into the machine. But surprisingly, we ourselves can also learn a lot of good life advice from this kind of thinking too. For instance, the model compresses a messy, diverse soup of inputs into a kind of pure, abstract token. You know, in life, when asking other people for advice, you will inevitably hear everything. And its opposite too. That is also a big soup of inputs. But try to look at all of them side by side, and you will find that they often share an underlying truth. This works, as is showcased by this incredible project too. And note that this work is not the end of anything. No! This is just a start, an early work at a nascent area. Two more papers down the line, and I really hope this is going to start folding my laundry and cooking my lunch. That would be amazing. What a time to be alive! And once again, this is not some proprietary nonsense. This is open knowledge and open models for free for all of us. Ha! That's just one of the many amazing papers they just dropped. If you are interested in hearing more, hopefully soon, subscribe and hit the bell. Thank you! Here you see me running the full Deepseek AI model through Lambda GPU cloud. 671 billion parameters running super fast and super reliably. This is insane. I love it. And I use it on a regular basis. Lambda provides you with powerful NVIDIA GPUs to run your own chatbots, and experiments. Seriously, try it out now at lambda.ai/papers or click the link in the description.
