Researchers trained an AI system on just three hours of motion-capture data to generate realistic, controllable basketball moves like dribbling, shooting, and catching. The technique outperforms older methods and works for other animal movements too.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. In computer games and all kinds of applications where we are yearning for realistic animation, we somehow need to tell our computers how the different joints and body parts of these virtual characters are meant to move around over time. Since the human eye is very sensitive to even the tiniest inaccuracies, we typically don't program these motions by hand, but instead we often record a lot of real time motion capture data in a studio and try to reuse that in our applications. Previous techniques have tackled quadruped control and we can even teach bipeds to interact with their environment in a realistic manner. Today we will have a look at an absolutely magnificent piece of work where the authors carved out a smaller subproblem and made a solution for it that is truly second to none. And this sub problem is simulating virtual characters playing basketball. Like with previous works, we are looking for realism in the movement and for games. It is also a requirement that the character responds to our controls well. However, the key challenge is that all we are given is three hours of unstructured motion capture data. That is next to nothing. And from this next to nothing, a learning algorithm has to learn to understand these motions so well that it can weave them together even when a specific movement combination is not present in this data. That is quite a challenge. Compared to many other works, this data is really not a lot. So I am excited to see what value we are getting out of these three hours. At first I thought we'd get only very rudimentary motions, and boy was I dead wrong on that one. We have control over this character and can perform these elaborate maneuvers and. And it remains very responsive even if we mash the controller like a madman producing these sharp turns. As you see, it can handle these cases really well. And not only that, but it is so well done. We can even dribble through a set of obstacles leading to a responsive and enjoyable gameplay. Now, about these dribbling behaviors. Do we get only one boring motion or not? Not at all. It was able to mine out not just one, but many kinds of dribbling motions and is able to weave them into other moves as soon as we interact with the controller. This is already very convincing, especially from just three hours of unstructured motion capture data. But this paper is just getting started. Now hold on to your papers because we can also shoot and catch the ball. Move it around. That is very surprising because it has looked at so little shooting data. Let's see. Yes, less than seven minutes. My goodness. And it keeps going. What I have found even more surprising is that it can handle unexpected movements, which I find to be even more remarkable given the limited training data. These crazy corner cases are typically learnable when they are available in abundance in the training data, which is not the case here. Amazing. When we compare these motions to a previous method, we see that both the character and the ball's movement is much more lively. For instance, here you can see that the phase function neural network pfnn, in short, almost makes it seem like the ball has to stick to the hand of the player for an unhealthy amount of time to be able to create these motions. It doesn't happen at all with the null technique. And remember, this null method is also much more responsive to the player's controls and thus more enjoyable not only to look at but to play with. This is an aspect that is hard to measure, but it is not to be underestimated in the general playing experience. Just imagine what this research area will be capable of not in a decade, but just two more papers down the line. Loving it now. At the start of the video I noted that the authors carved out a small use case which is training an AI to weave together basketball motion capture data in a manner that is both realistic and controllable. However, many times in research we look at a specialized problem and during that journey we learn general concepts that can be applied to other problems as well. That is exactly what happened here. As you see, parts of this technique can be generalized for quadruped control as well. This good boy is pacing and running around beautifully. And you guessed right, our favorite biped from the previous paper is also making an introduction. I am absolutely spellbound by this work and I hope that now you are too. Can't wait to see this implemented in newer games and other real time applications. What a time to be alive. This episode has been supported by weights and biases. In this post they show you how to use their system with Pytorch Lightning and decouple your science code from your engineering code and visualize your models. You can even try an example in an interactive notebook through the link in the video. Description Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to WeightsAndBiases for their long standing support and for for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
