This YouTube explainer (LARRUZ; Feb 2, 2021) presents frame interpolation (frame in-betweening) techniques—optical flow and learned neural-network methods—and examines applying learned interpolation to animate humanoid motion from sparse keyframes. The video highlights strong data efficiency: the method trained on the Ubisoft Laforge Animation Dataset (5 subjects, 77 sequences, ~4.5 hours, only 8 dancing sequences) can generate believable walks, runs, twists and dance moves. Evaluation uses normalized power spectrum similarity (NPSS), which correlates with human judgment; the new method outperforms prior techniques and the work was accepted to SIGGRAPH Asia. The video is narrated by Dr. Zsolnai and credits Felix Harvey (paper author) and Weights & Biases (sponsor).
Questions this video answers
February 2, 2021.
LARRUZ.
Dr. Zsolnai.
The Ubisoft Laforge Animation Dataset.
5 subjects, 77 sequences, about 4.5 hours of footage in total.
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Most people think that if we have a piece of camera footage that is a little choppy, then there is nothing we can do with it. We better throw it away. Is that true? No, not at all. Earlier, we discussed two potential techniques to remedy this common problem. The problem statement is in goes a choppy video, something happens, and then out comes a smooth and creamy video. This process is often referred to as frame interpolation or frame in betweening. And of course, it's easier said than done. If it works well, it really looks like magic, much like in the science fiction movies. So what are the potential somethings that we can use to make this happen? 1. Optical flow. This is an originally handcrafted method that tries to predict the motion that takes place between these frames. This can kind of produce new information, and I use this in these videos on a regular basis. But the output footage also has to be carefully inspected for unwanted artifacts, which are a relatively common occurrence. 2. We can also try to give a bunch of training data to a neural network and teach it to perform this frame in betweening. And if we do, the results are magnificent. We can do so much with this. But if we can do this for video frames, here is a crazy idea. How about a similar kind of in betweening for animating humanoids? That would really be something else, and it would save us so much time and work. Let's see what this new method can do in this area. The value proposition of this technique is as simple as it gets. We set up a bunch of keyframes. These are the transparent figures, and the neural network creates realistic motion that transitions from one stage to the next. 1. Look. It really seems to be able to do it all. It can perform twists and turns, brisk walks and runs, and you will see in a minute, even dance moves. Hmm. This in betweening for animating humanoid motion idea may not be so crazy after all. What's more, this could be super useful for artists working in the industry who can not only do all this, but they can also set up movement variations by moving the keyframes around spatially. Or we can even set up temporal variations to create different timings for the movement. Excellent. Of course, it cannot do everything. If we set up the intermediate stages in a way that uncommon motions would be required to fill in, we might end up with one of these failure cases. And all these results depend on how much training data we have with the kinds of motion we need to fill in. Let's have a look at A more detailed example, this smooth chap has been given lots of training data with dancing moves. And look, when we pull out these dance moves from his training data, he becomes a drunkard. So, talking about training data, how much motion capture footage was given to this algorithm? Well, it used the Ubisoft Laforge Animation Dataset. This contains five subjects, 77 sequences, and about 4 1/2 hours of footage in total. Wow, that is not that much. For instance, it only has eight movement sequences for dancing. That is not much at all. And we've already seen that the model can dance. That is some serious data efficiency, especially given that it can even climb through obstacles. So much knowledge has been extracted from so little data. It truly feels like we are living in a science fiction world. What a time to be alive. So when we write a paper like this, how do we compare the results to previous techniques? How can we decide which technique is better? Well, the Level 1 solution is a user study. We call some folks in, show them the footage, ask which one they liked best, the previous method or this one. That would work, but of course it is quite laborious. But fortunately there is a level 2 solution. And this level 2 solution is called normalized power spectrum similarity NPSS. In short, this is a number that we can produce with a computer. No humans are required, and it measures how believable these motions are. And the key of NPSS is that it correlates with human judgment. Or in other words, if this says that the technique is better, then it is likely that humans would also come to the same conclusion. So let's see those results. Here are the previous NPSS is subject to minimization. In other words, the lower the better. And let's see, the new method. Oh yes, it indeed outpaces the competition. So there is no wonder that this incredible paper was accepted to the Siggraph Asia conference. What does that mean exactly? If research were the Olympics, a SIGGRAPH or Siggraph Asia paper would be the gold medal. And this was Mr. Felix Harvey's first few papers. Huge congratulations. And as an additional goodie, it can create an animation of me when I lost my papers. And this is me when I found them. Do you have some more ideas on how we could put such an amazing technique to use? Let me know in the comments below. What you see here is a report of this exact paper we have talked about, which was made by weights and biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. If you work with learning algorithms on a regular basis, make sure to check out weights and biases. Their system is designed to help you organize your experiments and it is so good it could shave off weeks or even months of work from your projects and is completely free for all individuals, academics and open source projects. This really is as good as it gets and it is hardly a surprise that they are now used by over 200 companies and research institutions. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
