Károly Zsolnai-Fehér explains overfitting using a dog vs cat classifier. A model that memorizes training data fails on new examples, like studying a textbook instead of learning concepts. L1 and L2 regularization simplify decision boundaries, but too much regularization hurts performance.
Dear fellow scholars, this is two minute papers with in machine learning, we often encounter classification problems where we have to decide whether an image depicts a dog or a cat. We'll have an intuitive but simplified example where we imagine that the red dots represent dogs and the green ones are the cats. We first start learning on a training set, which means that we get a bunch of images that are points on this plane. And from these points we try to paint the parts of the plane red and green. These this way we can specify which regions correspond to the concept of dogs and cats. And after that we'll get new points that we don't know anything about. And we'll ask the algorithm, for instance, a neural network to classify these unknown images so it tells us whether it thinks that it is a dog or a cat. This is what we call a test set. We have had lots of fun with neural networks and deep Learning in previous 2 minute papers episodes. I've put some links in the description box. Check out, check them out. In this example, it is reasonably easy to tell that the reds roughly correspond to the left and greens to the right. However, if we just jumped on the deep learning hype train and don't know much about neural networks, we may get extremely poor results like this. What we see here is the problem of overfitting. Overfitting means that our beloved neural network does not learn the concept of dogs or cats. It just tries to adapt as much as possible to the training set as an intuition. Think of poorly made real life exams. We have a textbook where we can practice with exercises. So this textbook is our training set. Our test set is the exam. The goal is to learn from the textbook and obtain knowledge that proves to be useful at the exam. Overfitting means that we simply memorize parts of the textbook instead of obtaining real knowledge. If you are on page five and you see a bus, then the right answer is B. Memorizing patterns like this is not real learning. The worst case is if the exam questions are also from the textbook, because you can get a great grade just by overfitting. So this kind of overfitting has been a big looming problem in many education systems. Now the question is, which kind of neural network do we want? Something that works, like a lazy student or one that can learn many complicated concepts. If we are aiming for the latter, we have to combat overfitting, which is the bane of so many machine learning techniques. Now, there's several ways of doing that, but today we're going to talk about one possible solution by the name L1 and L2 regularization. The intuition of our problem is that the deeper and bigger neural networks we train, the more potent they are. But at the same time they get more prone to overfitting. The smarter the student is, the more patterns he can memorize. One solution is to hurl a smaller neural network at the problem. If this smaller version is powerful enough to take on the problem, we're good. A student who cannot afford to memorize all the examples is forced to learn the actual underlying concepts. However, it is very possible that this smaller neural network is not powerful enough to solve the problem. So we need to use a bigger one, but bigger network, more overfitting. Damn. So what do we do? And here is where L1 and L2 regularization comes to save the day. It is a tool to favor simpler models instead of complicated ones. The idea is that the simpler the model is, the better it transfers the textbook knowledge to the exam. And that's exactly what we're looking for. Here you see images of the same network with different regularization strengths. The first one barely helps anything. And as you can see, overfitting is still rampant. With a stronger L2 regularization, you see that the model is simplified substantially and is likely to perform better on the exam. However, if we add more regularization, it might be that we simplified the model too much and it is almost a same as a smaller neural network that is not powerful enough to grasp the underlying concepts of the exam. Keep your neural network as simple as possible, but not simpler. One has to find the right balance, which is an art by itself, and it shows that training deep neural networks takes a bit of expertise. It is more than just a plug and play tool that solves every problem by magic. If you want to play with the neural networks you've seen in this video, just click on the link in the description box. I hope you'll have at least as much fun with it as I had. Thanks for watching and for your generous support, and I'll see you next time.
