C4W2L09 Transfer Learning

If you're building a computer vision application, rather than training the weights from scratch from random initialization, you often make much faster progress if you download weights that someone else has already trained on a network architecture and use that as pre training and transfer that to a new task that you might be interested in. The computer vision research community has been pretty good at posting lots of data sets on the Internet. So if you hear of things like imagenet or mscoco or Pascal, types of datasets, these are the names of different datasets that people have posted online and that a lot of computer vision researchers have trained their algorithms on. Sometimes this training takes several weeks and might take many, many GPUs. And the fact that someone else has done this and gone through the painful high performance search process means that you can often download open source weights that took someone else many weeks or months to figure out and, and use that as a very good initialization for your own neural network and use transfer learning to sort of transfer knowledge from some of these very large public data sets to your own problem. Let's take a deeper look at how to do this. Let's start an example. Let's say you're building a cat detector to recognize your own pet cat. So according to the Internet, Tigger is a common cat name and Misty is another common cat's name. And let's say your cats are called Tigger and Misty. And there's also, you know, neither. So you have a classification problem with three classes. Is this picture Tigger or is it Misty or is it neither? And we'll ignore the case of both of you cats appearing in a picture. Now you probably don't have a lot of pictures of Tigger or Misty, so your training set will be small. So what can you do? I recommend you go online and download some open source implementation of a neural network and download not just the code, but also the weight. And there are a lot of networks, and there are a lot of networks you can download that have been trained on, for example, the ImageNet dataset, which has a thousand different classes. So the network might have a softmax unit that outputs one of a thousand possible classes. What you can do is then get rid of the softmax layer and create your own softmax unit that outputs Tigger or Misty or neither. And in terms of the network, I'd encourage you to think of all of these layers as frozen. So you freeze the parameters in all of these layers of the network and you would then just train the parameters associated with your softmax layer, which is A softmax layer with three possible outputs, you know, ticker, MSTI or neither. And by using someone else's pre trained weights, you might be able to get pretty good performance on this even with a small data set. Fortunately, a lot of deep learning frameworks support this mode of operation. And in fact, depending on the framework it might have things like trainable parameter equals zero. You might set that for some of these earlier layers in order to just say, you know, don't train those weights, um, or sometimes you have a parameter like freeze equals one. Uh, and these are different ways and different deep learning programming frameworks that let you specify whether or not to train the weights associated with a particular layer. And so in this case you would train only the softmax layer's weights, but freeze all of the earlier layers weights. One other neat trick that may help for some implementations is that because all of these early layers are frozen, uh, there is some fixed function that doesn't change because you're not changing it, you're not training it. That takes its input image X and maps it to some set of activations in that layer. So one other trick that could speed up training is you just pre compute that layer, um, the features really activations from that layer and just save them to disk. And what you're doing is you're using this fixed function in this first part of the neural network to take as input any input any image X and compute some feature vector for it. And then you're training a shallow softmax model from this feature vector to make a prediction. And so one step that could help your computation is you just pre compute that layer's activation for all the examples in your training set and save them to disk and then just train a softmax classifier on top of that. So the advantage of the save to this or the pre compute method, the save to this method is that you don't need to recompute those activations every time you take an epoch or take a pass through a training set. So this is what you do if you have a pretty small training set for your task. What if you have a larger training set? So one rule of thumb is if you have a larger label dataset. So maybe you just have a ton of pictures of ten Tigger, Misty, as well as, I guess, pictures of neither of them. One thing you could do is then freeze fewer layers. So maybe you freeze just these layers and then train these later layers. Although if the output layer has different classes, then you need to, you know, have your own output unit anyway of ticker Misty or neither. And there are a couple ways to do this. You could take the last few layers and there are a couple ways to do this. You could take the last few layers weights and just use that as initialization and do gradient descent from there. Or you could also blow away these last few layers and just, you know, use your own new hidden units and then your own final softmax output. So either of these methods could be worth trying, but maybe one pattern is if you have more data, the number of layers you freeze could be smaller and then the number of layers you train on top could be greater. And the idea is that if you have bigger data set, then maybe you have enough data not just to train a single softmax unit, but to train some, you know, modest size neural network that comprises the last few layers of this final network they end up using. And then finally, if you have a lot of data, one thing you might do is take this open source network and wait and use the whole thing just as initialization and train the whole network. Although again, if this was a thousand node softmax and you have just three outputs, you need your own softmax output to output the labels you care about. Um, but the more label data you have for your task, so the more pictures you have of Tigger MSTI and neither the more layers you could train. And in the extreme case, you could, um, use the weights you download just as initialization, so they would replace random initialization. And then you could do gradient descent training, updating all the weights and all the layers of the network. So that's transfer learning for the training of confnets in practice, because the open data sets on the Internet are so big and the weights you can download that someone else has spent weeks training has learned from so much data, you find that for a lot of computer vision applications, you just do much better if you download someone else's open source weights and use that as initialization for your problem. So in all the different disciplines, in all the different applications of deep learning, I think that computer vision is one where transfer learning is something that you should almost always do unless you actually have a very, very large, unless you have an exceptionally large dataset to train everything else from scratch yourself. But transfer learning is just very worth seriously considering unless you have an exceptionally large data set and a very large computational budget to train everything from scratch by yourself.