Károly Zsolnai-Fehér reviews an AI that isolates one person's voice from noise and other speakers by watching their face. Trained on nearly 300,000 videos, it works on anyone and handles bars, meetings, screaming kids, and overlapping dialogue without needing to know who the speaker is.
Dear fellow scholars, this is two minute papers with this is a neural network based technique that can perform audio visual separation. Before we talk about what that is, I will tell you what it is not. It is not what we've seen in the previous episode where we could select a pixel and listen to it. Have a look. And now let's try to separate the sound of the cello and see if it knows where it comes from. This one is different. This new technique can clean up an audio signal by suppressing the noise in a busy bar. Even if the source of the noise is not seen in the video, it can also enhance the voice of the speaker at the same time. Let's listen.
So the task is given the video. Any person who you see talking, their audio gets cleaned up and everything else gets suppressed. So the task is given the video, any person who you see talking, their audio gets cleaned up and everything else gets suppressed.
Or if we have a Skype meeting with someone in a lab or a busy office where multiple people are speaking nearby, we can also perform a similar speech separation, which would be a godsend for future meetings.
So we've been trying to train this network to input two embeddings and output three. Yeah, this is just an extra experiment for the paper. Hi guys. So we've been trying to train this network to input two embedding as an output three. Yeah, this is just an extra experiment for the pa.
And I think if you are a parent, the utility of this example needs no further explanation.
Okay Google. Okay Google, please navigate to work. OK Google.
I am not sure if I ever encountered the term screaming children in the abstract of an AI paper scene. So that one was also a first here. This is a super difficult task because the AI needs to understand what leap motions correspond to what kind of sounds, which is different for all kinds of languages, age groups and head positions. To this end, the authors put together a stupendously large dataset with almost 300,000 videos with clean speech signals. This data set is then run through a multi stream neural network that detects the number of human faces within the video, generates small thumbnails of them, and observes how they move over time. It also analyzes the audio signals separately, then fuses these elements together with the recurrent neural network to output the separated audio waveforms. A key advantage of this architecture and training method is that as opposed to many previous works, this is speaker independent. Therefore we don't need specific training data from the speaker. We want to use this on. This is a huge leap in terms of usability. The paper also contains an excellent demonstration of this concept by taking a piece of footage from Conan o' Brien's show where two comedians were booked for the same time slot and talk over each other. The result is a performance where it is near impossible to understand what they are saying, but with this technique we can hear both of them one by one, crystal clear. You see some results over here, but make sure to click the paper link in the description to hear the sound samples as well. Thanks for watching and for your generous support, and I'll see you next time.
