Rewrite Videos By Editing Text

Dear fellow scholars, this is two minute papers with the last few years have been an amazing ride when it comes to research works for creating facial reenactments for real characters. Beyond just transferring our gestures to a video footage of an existing talking head, controlling their gestures like video game characters and full body movement transfer are also a possibility. With Wavenet and and its many variants, we can also learn someone's way of speaking, write a piece of text and make an audio waveform where we can impersonate them using their own voice. So what else is there to do in this domain? Are we done? No, no, not at all. Hold on to your papers because with this amazing new technique, what we can do is look at the transcript of a talking head video, remove parts of it or add to it, just as we would edit any piece of text. And this technique produces both the audio and the matching video of this person uttering these words.

Check this out with Apple's stock price at $191.45 per share. Okay, the market closed today with Apple's stock price at $182.25 per share.

It works by looking through the video, collecting small sounds that can be used to piece together this new word that we've added to the transcript. The authors demonstrate this by adding the word fox to the transcript. This can be pieced together by the V which appears in the word viper and taking ox as a part of another word found in the footage. As a result, one can make the character say fox without ever hearing her uttering this word before. Then we can look for not only the audio occurrences for these sounds, but the video footage of how they are being said. And in the paper, a technique is proposed to blend these video assets together. Finally, we can provide all this information to a neural renderer that synthesizes a smooth video of this talking head. This is a beautiful architecture with lots of contributions. So make sure to have a look at the paper in the description for more details. And of course, as it is not easy to measure the quality of these results in a mathematical manner, a user study was made where they asked some fellow humans which is the real footage and which one was edited. You will see the footage edited by this algorithm on the right.

The dingo ate your baby. The dingo ate my baby. The dingo ate my baby.

And. Hmm, it's not easy to tell which one is which. And it also shows in the numbers which are not perfect, but they clearly show that the fake video is very often confused with the real one. Did you find any artifacts that give the trick away. Perhaps the sentence was said a touch faster than expected. Found anything else? Let me know in the comments below. The paper also contains tons of comparisons against previous works, so in the last few years the trend seems clear. The bar is getting lower, it is getting easier and easier to produce these kinds of videos, and it is getting harder and harder to catch them with our naked eyes. And now we can edit the transcript of what is being said, which is super convenient. I would like to note that AIs also exist that can detect these edited videos. With a high confidence. I put up the ethical considerations of the authors here. It is definitely worthy of your attention as it discusses how they think about these techniques. The motivation for this work was mainly to enhance digital storytelling by removing filler words, potentially flubbed phrases, or retiming sentences in talking head videos. There is so much more to it, so make sure to pause the video and read their full statement. Thanks for watching and for your generous support and I'll see you next time.