How to debug neural networks by checking if your gradient calculations are correct. Run it only during development, not training, because it's slow. When gradients disagree, check weights and biases separately by layer to find the bug. Remember to include regularization in your analytical gradients, and disable dropout while checking since it changes the cost randomly.
In the last video, you learned about gradient checking. In this video, I want to share you some practical tips or some notes on how to actually go about implementing this for your neural network. First, don't use grad check in training, only to debug. So what I mean is that computing d theta approx I for all the values of I, this is a very slow computation. So to implement gradient descent, you should use backprop to compute d theta and just use backprop to compute the derivatives. And it's only when you're debugging that you would compute this to make sure it's close to d theta. But once you've done that, then you would turn off the grad check. And don't run this during every iteration of gradient descent, because that's just much too slow. Second, if an algorithm fails grad check, look at the components, look at the individual components to try to identify the bug. So what I mean by that is if D theta approximately is very far from d theta, what I would do is look at the different values of I to see which are the values of d theta approximately. They're really very different than the values of d theta. So for example, um, if you find that the values of theta or d theta, they're very far off, all correspond to DBL for some layer or for some layers. But the components for DW are quite close, right? Remember, different components of theta correspond to different components of B, B and W. When you find this to be the case, then maybe you find that the the bug is in how you're computing db, the derivative, respective parameters B, um, and, and similarly vice versa. If you find that the values that are very far, you know the values from d theta approximately are very far from D theta. If you find that all those components came from DW or from DW in certain layer, then that might help you hone in on the location of the bug. This doesn't always let you identify the bug right away, but sometimes it helps you give you some guesses about how to where to track down the bug. Next, when doing grad check, remember your regularization term if you're using regularization. So if your cost function is j of theta equals 1 over m sum of your losses, um, and then plus this regularization term, right? Sum of l of wl Frobenius norm squared, then this is the definition of j and you should have that D theta is is gradient of j with respect to theta, including this regularization term. So just remember to include that term. Next, graph check. Doesn't Work with dropout because in every iteration dropout is randomly eliminating different subsets of the hidden units. There isn't a easy to compute cost function j that dropout is doing gradient descent on. It turns out that dropout can be viewed as optimizing some cost function j. But it's cost function j defined by summing over all x exponentially large subsets of nodes they could eliminate in any iteration. So the cost function J is very difficult to compute. Um, and you're just sampling the cost function every time you eliminate a different random subset of nodes when you use dropout. So it's difficult to use grad check to double check your computation with dropout. So what I usually do is implement grad check without dropout. So if you want, you can set heap prop in dropout to be equal to 1.0 and then turn on dropout and hope that's my implementation of dropout was correct. There are some other things you could do, like fix the pattern of nodes dropped and verify that drag chat for that pattern of units kill off is correct. But in practice I don't usually do that. So my recommendation is turn off dropout, use drag check to double check that your algorithm is at least correct without dropout, and then turn on dropout. Finally, this is a subtlety. It is not impossible. Rarely happens. But it's not impossible that your implementation of gradient descent is correct when W and B are close to zero, so at random initialization, but that as you run gradient descent and W and B become bigger, maybe your implementation of backprop is correct only when W and B is close to zero, but it gets more inaccurate when W and B become large. So one thing you could do, I don't do this very often, but one thing you could do is run drag check at random initialization and then train the network for so that W and B have some time to wander away from zero, from the small random initial values and then run grad check again after you've trained for some number of iterations. So that's it for gradient checking. And congratulations for coming to the end of this week's materials. In this week you've learned about how to set up your trained depth and test sets, how to analyze bias and variance, and what things to do if you have high bias versus high variance versus maybe high bias and high variance. Um, you also saw how to apply different forms of regularization like L2 regularization and dropout on your neural network. So some tricks for speeding up the training of your neural network and then finally gradient checking. So I think you've seen a lot in this week and you get to exercise a lot of these ideas in this week's programming exercise. So best of luck with that and I look forward to seeing you in the week. 2. Materials.
