AI Creates Facial Animation From Audio | Two Minute Papers #185

Dear fellow scholars, this is two minute papers withejir. You know, I think we're in a moment of history where probably the most important thing we need to do is to bring the country together. And one of the skills that I bring to bear. Somewhere over the rainbow.

This is my reality and this is the reality of my people for decades, for centuries.

This work is about creating facial animation from speech in real time. This means that after recording the audio footage of us speaking, we give it to a learning algorithm which creates a high quality animation depicting our digital characters uttering these words. This learning algorithm is a convolutional neural network which was trained on as little as three to five minutes of footage per actor and was able to generalize its knowledge from this training data to a variety of real world expressions and words. And if you think you've seen everything, you should watch until the end of the video as it gets better than that, because of two reasons. It not only takes audio input, but we can also specify an emotional state that the character should express when uttering these words.

You do not understand what it is to. Are those Eurasian footwear, cowboy chaps or jolly earth moving headgear? With tenure, Susie have all the more pleasure for yachting? Are those Eurasian footwear, cowboy chaps or jolly earth moving headgear? With tenure Susie'd have all the more pleasure. Are those Eurasian footwear cowboy chaps or jolly earthmoving headgear Number two.

And this is the best part. We can also combine this together with DeepMind's WaveNet, which synthesizes audio from our text input. It basically synthesizes a believable human voice and says whatever text we write down. And then that sound clip can be used with this technique to make a digital character say what we have written. So we can go from text to speech with Wavenet and put the speech onto a virtual actor with this work. The avocado is a pear shaped fruit with leathery skin, smooth edible flesh and a large stone. The avocado is a pear shaped fruit with leathery skin, smooth edible flesh and a large stone. The avocado is a pear shaped fruit with leathery skin, smooth edible flesh and a large stone. This way we get a whole pipeline that works by learning and does everything for us in the most convenient way. No actors needed for voiceovers, no motion capture for animations. This is truly incredible. And if you look at the left side, you can see that in their video there is some two minute papers action going on. How cool is that? Make sure to have a look at the paper to see the three way loss function the authors came up with to make sure that the results work correctly for longer animations. And of course, in research, we have to prove that our results are better than previous techniques to accomplish this. There are plenty of comparisons in the supplementary video, but we need more than that since these results cannot be boiled down to a mathematical theorem, that we need to prove that we have to do it some other way. And the ultimate goal is that a human being would judge these videos as being real with a higher chance than one made with a previous technique. This is the core idea behind the user study carried out in the paper. We bring in a bunch of people, present them with a video of the old and new technique without knowing which is which, and ask them which one they feel to be more natural, and the result was not even close. The new method is not only better overall, but I haven't found a single case, scenario or language where it didn't come out ahead. And that's extremely rare in research. Typically in a maturing field, new techniques introduce a different kind of trade off. For instance, less execution time, but at the cost of higher memory consumption is a classical case, but here it's just simply better in every regard. Excellent.

Listen up. The train yard warehouses is our main objective. Alexei, you and your men will do that. You have to go in and out very quick. I want you to get all the ammo and weapons that you can carry and come back as quickly as you can. You understand?

If you enjoyed this episode and would like to help us make better videos in the future, please consider supporting us on Patreon. You can pick up cool perks like watching these episodes in early access. Details are available in the video description. Beyond telling these important research stories, we are also using part of these funds to empower other research projects. I just made a small write up about this which is available on our Patreon page. The link is in the video description. Make sure to have a look. Thanks for watching and for your generous support and I'll see you next time. Sam.