A neural network converts mono audio into binaural sound using video frames as a guide. The method outperformed existing techniques in user studies and can separate individual instruments while creating spatial depth from a single audio track.
Dear fellow scholars, this is 2 minute papers with binaural or 2.5D audio means a sound recording that provides the listener.
With an amazing 3D ish sound sensation.
It produces a sound that feels highly.
Realistic when listened to through headphones and therefore using a pair is highly recommended for this episode. It sounds way more immersive than regular mono or even stereo audio signals, but also requires more expertise to produce and is therefore quite scarce on the Internet. Let's listen to the difference together.
We have not only heard sound samples here, but you could also see the accompanying video content which reveals the position of the players and the composition of.
The scene in which the recording is made. This sounds like a perfect fit for an AI to take a piece of mono audio and use this additional information to convert it to make it sound binaural.
This project is exactly about that where a deep convolutional neural network is used to look at both the video and the single channel audio content in our.
Footage footage and then predict what it.
Would have sounded like were it recorded.
As a binaural signal. Let's listen to a few results.
Sam.
The fact that we can use.
The visual content as well as the audio with this neural network also enables us to separate the sound of an.
Instrument within the mix. Let's listen.
To validate the results. The authors both used a quantitative mathematical way of comparing their results to the ground truth. And not only that, but they also carried out two user studies as well. In the first one, the ground truth.
Was shown to the users and they were asked to judge which of the two techniques were better.
In this study, this new method performed.
Better than previous methods, and in the second setup, users were asked to name the directions they hear the different instrument sounds coming from.
In this case, the new method outperformed.
The previous techniques by a significant margin, and if we keep progressing like this.
We may be at most a couple papers away from 2.5D audio synthesis that.
Sounds indistinguishable from the real deal. Looking forward to a future where we.
Can enjoy all kinds of video content.
With this kind of immersion. Thanks for watching and for your generous support and I'll see you next time.
