Dr. Zsolnai walks through AI that reconstructs speech from silent video of lip movements. The model trained on five speakers with about 20 hours of video each, and attention maps show it watches the mouth but also the forehead and eyebrows. Possible uses include silent video calls and helping people who've lost their voice.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When watching science fiction movies, we often encounter crazy devices and technologies that don't really exist, or sometimes ones that are not even possible to make. For instance, reconstructing sound from vibrations would be an excellent example of that and could make a great novel with the Secret Service trying to catch dangerous criminals. Except that it has already been done in real life research. I think you can imagine how surprised I was when I first saw this paper in 2014 that showcased a result where a camera looks at this bag of chips and from these tiny, tiny vibrations, it could reconstruct the sounds in the room. Let's listen.
Mary had a little lamb whose fleece was white as snow, and everywhere that Mary went, that lamb was sure to go.
Yes, this indeed sounds like science fiction. But 2014 was a long, long time ago. And since then we have a selection of powerful learning algorithms. And the question is, what's the next idea that sounded completely impossible a few years ago, which is now possible? Well, what about looking at silent footage from a speaker and trying to guess what they were saying? Check mark. That sounds absolutely impossible to me. Yet this new technique is able to produce the entirety of this speech after looking at a video footage of the lip movement.
Lets listen between the wavelength, the frequency and the speed of electromagnetic radiation. In fact, the products of the wavelength at the frequency is the speed.
Wow. So the first question is, of course, what was used as the training data? It used a data set with lecture videos and chess commentary from five speakers. And make no mistake, it takes a ton of data from these speakers, about 20 hours from each. But it uses video that was shot in a natural setting, which is something that we have in abundance on YouTube and other places on the Internet. Note that the neural network works on the same speakers it was trained on and was able to learn their gestures and leap movements remarkably well. However, this is not the first work attempting to do this, so let's see how it compares to the competition.
Set white with i7 soon. Set white with i7 soon. Set White with i7 soon. Set White with i7 soon.
The new one is very close to the true spoken sentence. Let's look at another one.
Eight field guns were captured in position. Eight field guns were captured in position.
Note that there are gestures, a reasonable amount of head movement and other factors at play, and the algorithm still does amazingly well. Potential applications of this could be video conferencing in zones where we have to be silent, giving a voice to people with the inability to speak. Due to aphonia or other conditions, or potentially fixing a piece of video footage where parts of the speech signal are corrupted. In these cases, the gaps could be filled with such a technique.
Look, let's look at a cell potential of 0.5 volts for an oxidation of bromide by permanganate. The question I have is what PK would cause this voltage? Would it be a ph?
Now let's have a look under the hood. If we visualize the activations within this neural network, we see that it found out that it mainly looks at the mouth of the speaker. That is of course not surprising. However, what is surprising is that the other regions, for instance around the forehead and eyebrows, are also important to the attention mechanism. Perhaps this could mean that it also looks at the gestures of the speaker and and uses that information for the speech synthesis. I find this aspect of the work very intriguing and would love to see some additional analysis on that. There is so much more in the paper. For instance, I mentioned giving a voice to people with aphonia, which should not be possible because we are training these neural networks for a specific speaker. But with an additional speaker embedding step, it is possible to pair up any speaker with any voice. This is another amazing work that makes me feel like we are living in a science fiction world. I can only imagine what we will be able to do with this technique. Two more papers down the line. If you have any ideas, feel free to speculate in the comment section below. What a time to be alive. This episode has been supported by Snap, Inc. What you Here is SnapML, a framework that helps you bring your own machine learning models to Snapchat's AR lenses. You can build augmented reality experiences for Snapchat's hundreds of millions of users and help them see the world through a different lens. You can also apply to SNAP's AR Creator residency Program with a proposal of how you would use Lens Studio for a creative project. If selected, you could receive a grant between 1 to $5,000 and work with Snap's technical and creative teams to bring your ideas to life. It doesn't get any better than that. Make sure to go to the link in the video description and apply for their residency program and try snapml today. Our thanks to Snap Inc. For helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
