DeepMind trained an AI to match sounds with videos without labels. It can tell if audio and video go together, pinpoint where a sound comes from in a frame, and find matching images for any audio clip. A guitar sound retrieves guitar images and highlights the strings as the source.
Dear fellow scholars, this is two minute.
Papers with this work is about creating an AI that can perform audio visual correspondence.
This means two really cool tasks. One when given a piece of video and audio, it can guess whether they match each other.
So I hope you guys enjoy the song and thanks for watching subscribing. Thank you, bye.
And 2 it can localize the source of the sounds heard in the video. And wait because this gets even better. As opposed to previous works here the entire network is trained from scratch and is able to perform cross modal retrieval. Cross modal retrieval means that we are able to give it an input sound and it will be able to find.
Pictures that will produce similar sounds or vice versa.
For instance, here the input is the.
Sound of a guitar. Note the loudspeaker icon in the corner.
And it shows us a bunch of either images or sounds that are similar.
Marvelous.
The training is unsupervised, which means that the algorithm is given a bunch of.
Data and learns without additional labels or instructions.
The architecture and results are compared to a previous work by the name, Look, Listen and Learn that we covered earlier in the series.
The link is available in the video description.
As you can see, both of them.
Run a convolutional neural network. This is one of my favorite parts about deep learning.
The very same algorithm is able to process and understand signals of very different kinds, video and audio. The old work concatenates this information and produces a binary yes no decision whether it thinks the two streams match. This new work tries to produce a number that encodes the distance between the video and the audio, kind of like the distance between two countries on a map. But both video and audio signals are embedded in the same map, and the.
Output decision always depends on how small or big this distance is. This distance metric is quite useful if.
We have an input video or audio signal. Choosing other video and audio snippets that have a low distance is one of the important steps that opens up the.
Door to this mag Cross modal retrieval. What a time to be alive. Some results are easy to verify, others may spark some debate.
For instance, it is quite interesting to see that the algorithm highlights the entirety.
Of the guitar string as a sound source. If you are curious about this mysterious.
Blue image here, make sure to have.
A look at the paper for an explanation. Now this is a story that we.
Would like to tell to as many people as possible.
Everyone needs to hear about this.
If you would like to help us.
With our quest, please consider supporting us on Patreon.
You can pick up some cool perks like getting early access to these videos.
Or deciding the order of upcoming episodes. Details are available in the video description.
Thanks for watching and for your generous.
Support and I'll see you next time.
Sam.
