Gradient checking verifies backprop by computing numerical gradients using finite differences and comparing them to your derivatives. Use eps around 1e-7, then check if the L2-normalized difference is below 1e-7 (good), around 1e-5 (inspect), or above 1e-3 (bugs).
Gradient checking is a technique that's helped me save tons of time and helped me find bugs in my implementations of backpropagation many times. Let's see how you could use it too, to debug or to verify that your implementation of backprop is correct. So your neural network will have some set of parameters, W1, B1, and so on up to WLPL. So to implement gradient checking, the first thing you should do is take all your parameters and reshape them in into a giant vector theta. So, so what you should do is take W, which is a matrix, and reshape it into a vector. You know, take all of these W's and reshape them into vectors and then concatenate all of these things, um, so that you have a giant vector theta. Giant vector parameters theta. So instead of a cost function J being a function of, you know, the W's and B's, you would now have the cost function J being just a function of theta. Next, with W and B ordered the same way, you can also take DW1, DB1 and so on, and reshape them into big giant vector D theta of the same dimension as theta, right? So same as before, reshape DW1 into a matrix. DB1 is already a vector V shape DWL. All the DWs which are matrices, remember DW1 has the same dimension as W1, DB1 has the same dimension of B1. So with the same sort of reshaping and concatenation operation, you can then reshape all of these derivatives into a giant vector d theta, which is the same dimension as theta. So the question is now, is d theta the gradient or the slope of the cost function j? So here's how you implement gradient checking and often abbreviate gradient checking to grad check. So first remember that J is now a function of the giant parameter theta, right? So you can also expand this out to J as a function of theta 1, theta 2, you know, theta 3 and so on, right? However, whatever is the dimension of this giant parameter vector theta. So to implement grad check, we, what you're going to do is implement a loop so that for each I, So for each component of theta, let's compute d theta approx I to B and we're gonna take a two sided difference. We'll take J of Theta Theta 1, Theta 2 up to Theta I and we're gonna nudge theta I to add epsilon to this. So just, you know, increase theta I by epsilon and keep everything else the same. And because it's taking a two sided difference. We're going to do the same on the other side with theta I. But now minus epsilon. And then all of the other elements of theta are left alone. And then we'll take this and we divide it by 2 theta. And what we saw from the previous video is that this should be approximately equal to d theta I, which, which is supposed to be the partial derivative of j with respect to, I guess theta I. If d theta I is, you know, the derivative of the cost function j. So what you're gonna do is you're gonna compute this for every value of I. And at the end you now end up with two vectors. You end up with this d theta approximately, um, and this is going to be the same dimension as d theta. And both of these are in turn the same dimension as theta. And what you want to do is check if these vectors are approximately equal to each other. So in detail. Well, how do you define whether or not two vectors are really reasonably close to each other? What I do is the following. Um, I would compute the Euclidean distance between these two vectors. D theta approx minus d theta. So just the L2 norm of this. Notice there's no square on top. So this is the sum of squares of elements of the differences. And then you take a square root to actually get the Euclidean distance. And then just to normalize by the length of these vectors, divide by d theta approx plus d theta. Just take the Euclidean length of these vectors and the row for the denominator is just in case any of these vectors are really small or really large, you know, the denominator turns this formula into a ratio. So when you implement this in practice, I use epsilon equals maybe 10 to the minus 7. So you know, one E minus 7. And with this range of epsilon, if you find that this formula gives you a value like 10 to the minus 7 or smaller, then that's great. Uh, it means that your derivative approximation is very likely correct. You know, this is just very small value. If it's maybe on the range of 10 to the minus 5, I would take a careful look. Maybe this is okay. But I might double check the components of this vector and make sure that none of the components are too large. And if some of the components of this difference are very large, there may be a bug somewhere. Um, and if this formula on the left is, you know, on the order of 10 to the minus 3, then I would worry, I would be much more concerned that maybe there's a bug somewhere. Uh, but you should really be getting values much smaller than 10 to the minus V. Oh, and if it's any bigger than 10 to the minus V, then I will would be quite concerned. I would seriously worry about whether or not there's a bug and I would Then you should then look at the individual components of theta to see, you know, if there's a specific value of I for which d theta proxi is very different from d theta I and use that to try to track down whether or not some of your derivative computations might be incorrect. And after some amount of debugging it finally it ends up being this kind of very small value, then you probably have a correct implementation. So when implementing neural network what often happens is I'll implement forward prop and then backprop and then I might find that this grad check gives a relatively big value and then I would suspect that it must be a bug go in debug, debug, debug, and after debugging for a while, if I find that it parses grad check with a small value, then you know, you can be much more confident that it's then correct. So you now know how gradient checking works. This has helped me find lots of bugs in my implementations of neural nets and I hope will help you too. In the next video I want to share with you some tips or some notes on how to actually implement gradient checking. Let's go on to the next video.
