A neural network learns to isolate individual instruments in video by connecting sound to specific pixels. Trained on 60 hours of musical performances, it can separate a cello from a guitar in the same frame, letting you mute or remix each one independently.
60 hours of musical performances.
No — the presenters state that no supervision is required.
Dear fellow scholars, this is two minute papers with this is a neural network based method that is able to show us the sound of pixels.
What this means is that it separates.
And localizes audio signals in videos.
The two keywords are separation and localization.
So let's take a look at these one by one.
Localization means that we can pick a.
Pixel in the image and you and it shows us the sound that comes from that location.
And the separation part means that ideally.
We only hear that particular sound source.
Let's have a look at an example.
Here's an input video.
And now let's try to separate the sound of the cello and see if.
It knows where it comes from. Same with the guitar.
Now for a trickier question.
Even though there are sound reverberations off the walls, but the walls don't directly emit sound themselves. So I am hoping to hear nothing.
Now lets see. Flat signal.
Great.
So how does this work?
It is a neural network based solution.
That has watched 60 hours of musical.
Performances to be able to pull this off.
And it learns that a change in.
Sound can often track back to a.
Change in the video footage as a.
Musician is playing an instrument.
As a result.
Get this, no supervision is required.
This means that we don't need to label this data or in other words.
We don't need to specify how each pixel sounds.
It learns to infer all this information from the video and sound signals.
By itself, this is huge. And otherwise, just imagine how many work.
Hours that would require to annotate all this data. And another cool application is that if.
We can separate these signals, then we.
Can also independently adjust the sound of these instruments.
Have a look.
Now clearly it is not perfect as.
Some frequencies may bleed over from one instrument to the other.
And there are also other methods to.
Separate audio signals, but this particular one does not require any expertise.
So I see a great value proposition there.
If you wish to create a separate.
Version of a video clip and use.
It for karaoke, or just subtract the.
Guitar and play it yourself, I would look no further.
Also, you know the drill.
This will be way better a couple.
Papers down the line. So what do you think? What possible applications do you envision for this? Where could it be improved?
Let me know below in the comments.
Thanks for watching and for your generous.
Support and I'll see you next time.
