Dr. Zsolnai breaks down how neural networks build complexity from simple parts. Early layers detect car wheels and windows, later layers combine them into a car detector. One network learned a dog-head detector that recognizes the same dog from any angle.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This paper is not your usual paper, but it does something quite novel. It appeared in the Distilled Journal, one of my favorites, which offers new and exciting ways of publishing beautiful but unusual works aiming for exceptional clarity and readability. And of course, this new paper is no different. It claims that despite the fact that these neural network based learning algorithms look almost unfathomably complex inside, if we look under the hood, we can often find meaningful algorithms in there. Well, I am quite excited for this, so sign me up. Let's have a look at an example. At the risk of oversimplifying the explanation, we can say that a neural network is given as a collection of neurons and connections. If you look here, you can see the visualization of three neurons. At first glance, they look like an absolute mess. But don't they? Well, kind of. But upon closer inspection, we see that there is quite a bit of structure here. For instance, the upper part looks like a car window. The next one resembles a car body. And the bottom of the third neuron clearly contains a wheel detector. However, no car looks exactly like these neurons. So what does the network do with all this? Well, in the next layer, the neurons arise as a combination of neurons in the previous layers, where we cherry pick parts of each neuron that we wish to use. So here with red, you see that we are exciting the upper part of this neuron to get the window, use roughly the entirety of the middle one and use the bottom part of the third one to assemble this. And now we have a neuron in the next layer that will help us detect whether we see a car in an image or not. So cool. I love this one. Let's look at another example. Here you see a dog head detector, but it kind of looks like a crazy Picasso painting where he tried to paint a human from not one angle like everyone else, but from all possible angles on one image. But this is a neural network, so why engage in this kind of insanity? Well, if we have a picture of a dog, the orientation of the head of the dog can be anything. It can be a frontal image. Look from the left to right, right to left, and so on. So this is a pose invariant dog head detector. What this means is that it can detect many different orientations. And look here, you see that it gets very excited by all of these good boys. I think we even have a squirrel in here. Good thing this is not the only neuron we have in the network to make a decision. I hope that it already shows that this is truly an ingenious design. If you have a look at the paper in the video description, which you should absolutely do, you'll see exactly how these neurons are built from the neurons in the previous layers. The article contains way, way more than this. You'll see a lot more doc snouts, curve detectors and even a follow up article that you can have a look at and even comment on before it gets finished. A huge thank you to Chris Ola who devotes his time away from research and uses his own money to run this amazing journal. I cannot wait to cover more of these articles in future episodes, so make sure to subscribe and hit the bell icon to never miss any of those. So finally we understand a little more how neural networks do all these amazing things they are able to do. What a time to be alive. This episode has been supported by Weights and Biases. Here they show you how you can visualize the training process for your boosted trees with XGBOOST using that tool. If you have a closer look you'll see that all you need is one line of code. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weightsand Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
