Sam explains a method to remove convolutional filters from neural networks while keeping accuracy loss below a budget you set. The technique can prune 25% of filters with just 0.3% accuracy drop, or over 75% with 3% loss, making models faster and smaller for deployment.
Questions this video answers
The maximum acceptable accuracy drop (accuracy-loss budget).
Around 25% of the filters.
More than 75% of the filters.
Chapters
Dear fellow scholars, this is two minute papers with when we are talking about deep learning, we are talking about neural networks that have tens, sometimes hundreds of layers and hundreds of neurons within these layers. This is an enormous number of parameters to train and clearly there should be some redundancy, some duplication in the information within. This paper is trying to throw out many of these neurons of the network without affecting its accuracy too much. This process we shall call pruning, and it helps creating neural networks that are faster and smaller. The accuracy term I used typically means a score on a classification task. In other words, how good this learning algorithm is in telling what an image or video depicts. This particular technique is specialized for pruning convolutional neural networks where the neurons are endowed with a small receptive field and are better suited for images. These neurons are also commonly referred to as filters. So here we have to provide a good mathematical definition of a proper pruning. The authors proposed a definition where we can specify a maximum accuracy drop that we deem to be acceptable, which will be denoted with the letter B in a moment. And the goal is to prune as many filters as we can without going over the specified accuracy loss budget. The pruning process is controlled by an accuracy and efficiency term, and the goal is to have some sort of balance between the two. To get a more visual understanding of what is happening. Here, the filters you see outlined with the red border are kept by the algorithm and the rest are discarded. As you can see, the algorithm is not as trivial as many previous approaches that just prune away filters with weaker responses. Here you see the table with the B numbers. Initial tests reveal that around a quarter of the filters can be pruned with an accuracy loss of 0.3%. And with a higher B, we can prune more than 75% of the filters with a loss of around 3%. This is incredible. Image segmentation tasks are about finding the regions that different objects inhabit. Interestingly, when trying the pruning for this task image, it not only introduces a minimal loss of accuracy, in some cases, the pruned version of the neural network performs even better. How cool is that? And of course, the best part is that we can choose a trade off that is appropriate for our application. For instance, if we are looking for a light cleanup, we can use the first option at a minimal penalty. Or if we wish to have a tiny, tiny neural network that can run on a mobile device, we can look for the more heavy handed approach by sacrificing just a tiny bit more accuracy and we have everything in between. There is plenty more validation for the method in the paper. Make sure to have a look. It is really great to see that new research works make neural networks not only more powerful over time, but there are efforts in making them smaller and more efficient at the same time. Great news indeed. Thanks for watching and for your generous support and I'll see you next time. Sam.
