An AI learns to simulate realistic tennis player motions from raw video alone, without motion capture hardware. It uses physics simulation to clean up noisy pose estimates and can reproduce player-specific styles like left or right-handed backhands, hitting targets with up to 100% accuracy after training on 4.5 hours of footage.
Wow. I can't believe it. This new AI can look at a real tennis player and put them in a computer game where we can play with them. This looks absolutely amazing. And doing this the hard way has so far been absolutely impossible. But what about the easy way? Exciting as this may seem, perhaps there is nothing new here. You see, we can already do something like this through motion capture. What is that? Well, you put a human in a studio, attach these little balls to them and record their motion, then transfer this motion to a virtual character. However, this paper has an incredible twist. And that twist is that this can do the same, but without the balls. No cameras, gyroscopes or any additional machinery is required. Just the RAW video feed. And the learning happens from that. That sounds incredible, but how? Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Well, there are previous works that tried to do something similar, but the precision of these motions was almost never satisfactory. Especially for tennis, where almost hitting a ball is not nearly good enough. And I am very proud to say that all three of these papers have been showcased here on Two Minute Papers before. But this new technique, this one can do it even better. First we give it the RAW video feed of the players and then it will try to estimate their motion. So, are we done? Well, unfortunately not. Not even close. Look, the motion is still way too jerky and imprecise to be used in practice. It is also full of artifacts. So what now? Well, now comes the magic. Here a computer simulation is built that tries to reproduce this motion. But this one cannot be jerky because it is within a video game where the laws of physics are intact. In reality, people don't move like that. And with this simulation, this is looking so much better. From now on, we can take a target position that we start from and simulate many, many motion types. Then we grab a controller and perform these serves. Topspin, backspin, hits, slices, you name it. And they look absolutely incredible. Yes, I hear you asking, Karoj, are you sure that this is a simulation? Well, if you look here, here is the proof. Yes, simulations today are getting so advanced that we need to see these 3D models to be absolutely sure that this is indeed a computer simulation. Wow. And it gets better. It. Here is the interesting part. This is not just a copying machine. Although just copying the motion from these videos would also be quite a feat given the occlusions and tiny but super important wrist motions that are really hard to see from the footage. But with this system, we can even highlight a region where we would like to hit the ball, and the motions will be synthesized accordingly. We can run the same simulation with a number of different target locations, and all of the synthesized motions will be believable and accurate. Loving it. Or we can also ask the player to hit the ball from the same point from different starting positions. And it will work not only for the easier cases, but in more difficult scenarios too, where you can barely catch that ball so good. And now hold on to your papers, fellow scholars, because it can not only do all this, but it can even learn the individual style of famous tennis players. Right and left handed players work too, and it can simulate a one or two handed backhand shot, depending on the style of the player at hand. And if that is possible at all, it gets even better. If we can simulate one virtual human, why not make a game by simulating two playing against each other? I have to say, if I look closely, I can still see that this is a simulation. But if you squint a little, it's easy to think that this is real footage. That is an incredible achievement both in the graphics and in the animation departments. Bravo. And styles are particularly interesting here because if it would have only one style, two players playing the same style would not look nearly as interesting as this footage with one left and one right handed player. What I particularly like is that there is barely any foot sliding, a problem that you see in almost every single paper in this area. They usually compete on how little foot sliding there is, because no foot sliding at all is almost unheard of. Wow. Also, almost none of the jitters from the original motions are there in this footage. The foot sliding here is less than an inch and the jitter is almost non existent. All gone. And if we remove just one small part of this new proposed system, things can go really bad. For instance, if the wrist motions are captured with just a bit of inaccuracy, the hits will go kind completely wrong. You have to get everything nearly perfectly for this to work. And the scientists absolutely nailed it. So how many hundreds of hours did the AI had to sift through to learn all this? Well, it was not in the order of hundreds of hours, but get this, only four and a half hours of video footage. Goodness, I don't even know what to say. When using the full system, the hit rates can reach up to 100%, which sounds almost impossible to me. They absolutely knocked this out of the court with this paper. And thus it was published at Siggraph, perhaps the most prestigious conference in computer graphics. In my view, it is like winning the Olympic gold medal of computer graphics research and I have to say it is very well deserved. Bravo. This might be the future of video games experiment tracking, model evaluation and production monitoring for your deep learning projects and LLM apps. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
