DeepMind's AI Learns Audio And Video Concepts By Itself | Two Minute Papers #184

Dear fellow scholars, this is Two Minute Papers with In our earlier episodes when it came to learning techniques, we almost always talked about supervised learning. This means that we give the algorithm a bunch of images and some additional information. For instance, that these images depict dogs or cats. Then the learning algorithm is exposed to new images that it had never seen before and has to be able to classify them correctly. It is kind of like a teacher sitting next to a student providing supervision. Then the exam comes with new questions. This is supervised learning. And as you have seen from more than 180 episodes of 2 Minute Papers, there is no doubt that this is an enormously successful field of research. However, this means that we have to label our datasets, so we have to add some additional information to every image we have. This is a very laborious task, which is typically performed by researchers or through crowdsourcing, both of which takes a lot of funding and hundreds of work hours. But if we think about it, we have a ton of videos on the Internet. You always hear these mind melting new statistics on how many hours of video footage is uploaded to YouTube every day. Of course, we could hire all the employees in the world to annotate these videos frame by frame to tell the algorithm that this is a guitar and this is an accordion or a keyboard, and we would still not be able to learn on most of what's uploaded. But it would be so great to have an algorithm that can learn on unlabeled data. However, there are learning techniques in the field of unsupervised learning, which means that the algorithm is given a bunch of images or any media and is instructed to learn on it without any additional information. There is no teacher to supervise the learning. The the algorithm learns by itself, and in this work, the objective is to learn both visual and audio related tasks in an unsupervised manner. So, for instance, if we look at this layer of the visual subnetwork, we'll find neurons that get very excited when they see, for instance, someone playing an accordion. And each of the neurons in this layer belong to different object classes. I surely have something like this for papers. And here comes the karoi Goes Crazy Part one. This technique not only classifies the frames of the videos, but it also creates semantic heatmaps which show us which part of the image is responsible for the sounds that we hear. This is insanity. To accomplish this, they ran a vision subnetwork on the video part and a separate audio subnetwork to learn about the sounds. And at the last step, all this information is fused together to obtain Karoj Goes Crazy Part two, which this makes the network able to guess whether the audio and the video stream correspond to each other. It looks at a man with a fiddle, listens to a sound clip, and will say whether the two correspond to each other. Wow. The audio subnetwork also learned the concept of human voices, the sound of water, wind music, live concerts, and much, much more. And the answer is yes. It is remarkably close to human level performance on sound classification. And all this is provided by the two networks that were trained from scratch and no supervision is required. We don't need to annotate these videos. Nailed it. And please don't get this wrong. It's not like DeepMind has suddenly invented unsupervised learning. Not at all. This is a field that has been actively researched for decades. It's just that we rarely see really punchy results like these ones here. Truly incredible work. If you enjoyed this episode and you feel that eight of these videos a month is worth a dollar, consider supporting us on Patreon. Details are available in the video Description. Thanks for watching and for your generous support, and I'll see you next time.