Researchers can reconstruct audio from high-speed video of vibrating surfaces. A bag of chips, plastic bag, or water captured on camera becomes a microphone. The technique could detect earthquakes, spot fake dubbing, or recover censored speech.
Dear fellow scholars, this is two minute papers with the work we are going to discuss today is about visual microphones. What does this mean exactly? The influence of sound creates surface vibrations in many objects such as plants or a bag of chips, foil containers, water and even bricks. They thereby work as visual microphones. Now hold on to your chairs because this algorithm can reconstruct all the audio data from video footage of vibrations. What this means is that if someone outside of your house pointed a high speed camera at a bag of chips, when you start talking in your room, he will be able to guess what you said by only seeing the vibrations of the bag of chips. In the following example, you'll see a recorded footage of the bag, but the movement is so subtle that your naked eye won't see any of it. First you'll hear the speech of a person recorded in the house, then the reconstruction from only the visual footage of the bag. Mary had a little lamb whose fleece was white as snow, and everywhere that Mary went, that lamb was sure to go. And this is what we were able to recover from high speed video filmed from outside behind soundproof glass. This is just unbelievable. Here is another example with a plastic bag where you can see the movement caused by the sound waves. The paper is very detailed and rigorous. This is definitely one of the best research works I've seen in a while. The most awesome part of this is that this is not only an excellent piece of research, it is also a great product. And note that this problem is even harder than one would think since the frequency response of various objects can be quite different, which means that every single object vibrates a bit differently when hit by the same sound waves. You can see a great example of these responses from bricks, water and many others here. What it will be used for is shrouded in mystery for the moment. Even though I think this work provides fertile grounds for new conspiracy theories, the authors don't believe it is suitable to use for surveillance. Someone argued that it may be useful for early earthquake detection, which is an awesome idea. Also, maybe it could be used to detect video redubbing and recovering beeped out speech from videos. And I'm sure there will be many other applications. The authors also have a follow up paper on estimating material properties by looking at how objects vibrate. Awesome. Do you have any potential applications in mind? Let me know in the comment section. And there's also a fantastic TED Talk and paper video on the topic that you can find in the description box alongside with a great Reddit discussion link. I urge you to check all of these out. The videos are narrated by Abe Davis, who did a great job at explaining their concepts. Thanks for watching and I'll see you next time.
