Researchers built a system that watches tennis broadcasts and generates realistic video game animations of matches. The AI extracts player poses and ball physics from footage, then uses strategy patterns and image translation to fill in missing details and create playable sequences.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Approximately a year ago we talked about an absolutely amazing paper by the name Vidtogame in which we could grab a controller and become video game characters. It was among the first introductory papers to tackle this problem. And in this series we always say that two more papers down the line and it will be improved significantly. So lets see what's in store. And this time just one more paper down the line. This new work offers an impressive value proposition which is to transform a real tennis match into a realistic looking video game that is controllable. This includes synthesizing not only movements, but also what effect the movements have on the ball as well. So how do we control this? And now hold on to your papers because we can specify where the next shot would land and with just one click, for instance, we can place this red dot here. And now just think about the fact that this doesn't just change where the ball should go, but the trajectory of the ball should be computed using a physical model and the kind of shot the tennis player has to perform for the resulting ball trajectory to look believable. This physical model even contains the ball's spin velocity and the magnus effect created by the spin. The entire chain of animations has to be correct. And that's exactly what happens here with Blue. We can also specify the position the player has to await in to hit the ball next. And these virtual characters don't just look like their real counterparts, they also play like them. You see, the authors analyzed the playstyle of these athletes and built a heat map that contains information about their usual shot placements for the forehand and and backhand shots separately, the average velocities of these shots, and even their favored recovery positions. If you have a closer look at the paper, you will see that they not only include this kind of statistical knowledge into their system, but they really went the extra mile and included common tennis strategies as well. So how does it work? Let's look under the hood. First. It looks at broadcast footage from which annotated clips are extracted that contain the movement of these players. If you look carefully, you see this red line on the spine of the player. And some more. These are annotations that tell the AI about the pose of the players. It builds a database from these clips and chooses the appropriate piece of footage for the action that is about to happen. Which sounds great in theory, but in a moment you will see that this is not nearly enough to produce a believable animation. For instance, we also need a rendering step which has to adjust this footage to the appropriate perspective, as you see here. But we have to do way more to make this work look without additional considerations, we get something like this. Not good. So what happened here? Well, given the fact that the source datasets contain matches that are several hours long, they therefore contain many different lighting conditions. With this, visual glitches are practically guaranteed to happen. To address this, the paper describes a normalization step that can even these changes out. How well does this do its job? Let's have a look. This is the unnormalized case. This short sequence appears to contain at least four of these glitches, all of which are quite apparent. And now let's see the new system after the normalization step. Yep, that's what I'm talking about. But these are not the only considerations the authors had to take to produce these amazing results. You see, oftentimes quite a bit of information is missing from these frames. Our seasoned fellow scholars know not to despair because we can reach out to image inpainting methods to address this. These can fill in missing details in images with sensible information. You can see Nvidia's work from two years ago that could do this reliably for a great variety of images. This new work uses a learning based technique called Image to Image Translation to fill in these details. Of course, the advantages of this new system are visible right away, and so are its limitations. For instance, temporal coherence could be improved, meaning that the tennis rackets can appear or disappear from from one frame to another. The sprites are not as detailed as they could be. But none of this really matters. What matters is that now what's been previously impossible is now possible. And two more papers down the line, it is very likely that all of these issues will be ironed out. What a time to be alive. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI generic, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus they are the only cloud service with 48 gigabyte RTX 8000. Join researchers at organizations like Apple, MIT and Caltech Institute using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support. And I'll see you next time. Thanks for watching.
