This content was created by LARRUZ. This content was uploaded by LARRUZ. This is a YouTube video titled "Distilling Neural Networks | Two Minute Papers #218" (created 2018-01-06). It is a single-speaker Two Minute Papers episode in which the host explains research on distilling neural networks into decision trees to improve interpretability. The video covers motivation (neural nets are hard to interpret), the proposed approach (use a trained network to generate additional training data and train decision trees on it), demonstrations on MNIST and Connect 4, and results (these distilled trees outperform traditional decision trees, are only slightly worse than the original networks, are faster, and provide clearer explanations). The host also references related work on tree boosting and invites viewers to follow links in the description.
Chapters
Dear fellow scholars, this is two minute papers with since the latest explosion in AI research, virtually no field of science remains untouched by neural networks. These are amazing tools that help us solve problems where the solutions are easy to identify but difficult to explain. For instance, we all know a backflip when we see one, but mathematically defining all the required forces, rotations and torque, is much more challenging. Neural networks excel at these kinds of tasks putting provided that we can supply them a large number of training samples. If we peek inside these neural networks, we'll see more and more layers and more and more neurons within these layers as years go by. The final decision depends on what neurons are activated by our inputs. They are highly efficient. However, trying to understand how a decision is being made by these networks is going to be a fruitless endeavor. This is especially troublesome when the network gives us a wrong answer that that we, without having access to any sort of explanation, may erroneously accept without proper consideration. This piece of work is about distillation, which means that we take a neural network and try to express its inner workings in the form of a decision tree. Decision trees take into consideration a series of variables and provide a clear roadmap towards a decision based on them. For instance, they are useful in using the age and amount of free time of people to try to guess whether they are likely to play video games or decide deciding who should get a loan from the bank based on their age, occupation and income. Yeah, this sounds great. However, the main issue is that decision trees are not good substitutes for neural networks. The theory says that we have a generalization versus interpretability trade off situation, which means that trees that provide us good decisions overfit the training data and generalize poorly, and the ones that are easy to interpret inaccurate. So in order to break out of this trade off situation, a key idea of this piece of work is to ask the neural network to build a decision tree by taking an input dataset for training, trying to generate more training data that follows the same properties and feed all this to the decision tree. Here are some results for the classical problem of identifying digits in the MNIST dataset. As each decision is meant to cut the number of output options in half each, it shows really well that we can very effectively perform the classification in only four decisions from a given input. And not only that, but it also shows what it is looking for. For instance, here we can see that the final decision before we conclude whether the Input number is a 3 or 8, it looks for the presence of a tiny area that joins the ends of the three to make an eight. A different visualization of the Connect 4 game dataset reveals that the neural network quickly tries to distinguish types of one where the players start playing on the inner and one with the outer region of the board. It is shown that these trees perform better than traditional decision trees. What's more, they are only slightly worse than the corresponding neural networks, but can explain their decisions much more clearly and are also faster. In summary, the rate of progress in machine learning research is truly insane these days, and I am all for papers that try to provide us a greater understanding of what is happening under the hood. I am loving this idea. In particular, we had an earlier episode on how to supercharge these decision trees via tree boosting. If you're interested in learning more about it, the link is available in the video description. Thanks for watching and for your generous support, and I'll see you next time.
