Short YouTube video (Two Minute Papers) describing a paper that retimes and alters real-person video to match arbitrary audio using recurrent neural networks, a mouth-texture synthesizer, pose-matching, and final retiming; demonstrates results on Barack Obama footage and notes improvements from WaveNet audio and other corrections (jaw correction, pose alignment).
Dear fellow scholars, this is two minute papers. With this work is doing something truly remarkable. If we have a piece of audio of a real person speaking and a target video footage, it will retime and change the video so that the target person appears to be uttering these words.
Thursday and one thing I told them is that they're not alone. The American people and people all over the world are standing with them. And we always will. To help families refinance their homes, to invest in things like high tech, manufacturing, clean energy, and the infrastructure that creates good new jobs. Not to mention the job training that helps folks earn new skills to fill those jobs. The results are clear. America's businesses have created 14.5 million new jobs over 75 straight months.
Whoa. This is different from what we've seen a few episodes ago where scientists at Nvidia worked on synthesizing lip sync geometry for digital characters, solely relying on audio footage. The results were quite amazing. Have a look. Listen up. The train yard warehouses is our main objective. Alexei, you and your men will do that. You have to go in and out very quick. This was great for animating digital characters when all we have is sound, but this time around, we are interested in reanimating the footage of real existing people. The prerequisite to do this with a learning algorithm is to have a ton of data to train on which we have in our possession, as there are many hours of footage of the former President speaking during his weekly address. This is done using a recurrent neural network. Recurrent neural networks are learning algorithms where the inputs and outputs can be sequences of data. So here in the first part, the input can be a piece of audio with the person saying something, and it is able to synthesize the appropriate mouth shapes and their evolution over time to match the audio. The next step is creating an actual mouth texture from this rough shape that comes out from the learning algorithm, which is then used as an input to the synthesizer. Furthermore, the algorithm is also endowed with an additional pose matching module to make sure that the synthesized mouth texture aligns with the posture of the head properly. The final retiming step makes sure that the head motions follow the speech correctly. If you have any doubts whether this is required, here are some results. With and without the retiming step.
I grew up without my father around.
Without retiming, he moves randomly and appears unnatural.
While I wonder what my life would have been like if he had been a greater presence, I've also tried extra hard to be a good dad for my own daughters. Like all dads, I worry about my girl's safety all the time.
You can see that this indeed substantially enhances the realism of the final footage. Even better when combined with Google DeepMind's WaveNet. Given enough training data, we could skip the audio footage altogether and just write a piece of text making Obama or someone else say what we have written. The Blue lagoon is a 1980American romance and adventure film directed by Randall Kleiser. Aspects of the sublime in English poetry and painting, 1770-1850 There are also a ton of other details to be worked out. For instance, there are cases where the mouth moves before the person starts to speak, which is to be taken into consideration. The dreaded ums and ahs are classical.
Examples of that small confines of the legal community. I think it's real important to keep the focus on the broader world out there and see that for a lot of kids, the doors that have been open to me aren't open to them.
There's also an important jaw correction step and more this is a brilliant piece of work with many non trivial decisions that are described in the paper. Make sure to have a look at it for details. As always, the link is available in the video description. The results are also compared to the face to face paper from last year that we also covered in the series. It is absolutely insane to see this rate of progress over the lapse of only one year. If you have enjoyed this episode and you feel that eight of these videos a month is worth a dollar, please consider supporting us on Patreon. You can pick up some really cool perks there and it is also a great deal of help for us to make better videos for you in the future. Earlier, I also wrote a few words about the changes we were able to make because of your amazing support. Details are available in the description. Thanks for watching and for your generous support and I'll see you next time.
