Lecture (Andrew Ng) describing how to compute derivatives (slopes) of common neural-network activation functions needed for backpropagation. Covers: sigmoid (g'(z)=g(z)(1-g(z)), equivalently a(1-a) when a=g(z)); tanh (g'(z)=1-(tanh z)^2, equivalently 1-a^2 when a=tanh(z)); ReLU (g(z)=max(0,z), g'(z)=0 for z<0, 1 for z>0, undefined at z=0 — subgradient used in practice); leaky ReLU (g(z)=max(0.01z,z), g'(z)=0.01 for z<0, 1 for z>0). Ends with implementation advice for gradient descent.
Questions this video answers
g'(z) = g(z) (1 - g(z)); equivalently, if a = g(z), then g'(z) = a (1 - a).
g'(z) = 1 - (tanh z)^2; equivalently, if a = tanh(z), then g'(z) = 1 - a^2.
g'(z) = 0 for z < 0; g'(z) = 1 for z > 0; technically undefined at z = 0 (a subgradient may be used in practice).
For g(z)=max(0.01z, z): g'(z) = 0.01 for z < 0; g'(z) = 1 for z > 0; technically undefined at z = 0.
Chapters
When you implement backpropagation for your neural network, you need to really compute the slope or the derivative of the activation functions. So let's take a look at our choices of activation functions and how you can compute the slope of these functions. Here's the familiar sigmoid activation function. And so, for any given value of z, maybe this value of Z, this function will have some slope or some derivative corresponding to. If you draw a little line there, you know the height over width of this little triangle here. So if G is the sigmoid function, then the slope of the function is d DZ g. And so we know from calculus that this is the slope of G at z. And if you are familiar with calculus and know how to take derivatives, if you take the derivative of the sigmoid function, it is possible to show that it is equal to this formula. And again, I'm not going to do the calculus steps, but if you're familiar with calculus, feel free to pause the video and try to prove this yourself. And so this is equal to just g of Z times 1 minus G of Z. So let's just sanity check that this expression makes sense. First, if Z is very large, let's say Z is equal to 10, then G will be close to 1. And so the formula we have on the left tells us that d DZ g of Z does be close to G of Z, which is equal to one times one minus one, which is therefore very close to zero. And this is indeed correct, because when Z is very large, the slope is close to zero. Conversely, if z is equal to minus 10, so say it's way out there, then G of z is close to zero. So the formula on the left tells us d DZ g of Z will be close to G of z, which is 0 times 1 minus 0. And so this is also very close to 0, which is correct. Finally, if z is equal to 0, then g of z is equal to 1/2. That's a sigmoid function right here. And so the derivative is equal to 1/2 times 1 minus 1/2, which is equal to 1/4. And that actually is turns out to be the correct value of the derivative or the slope of this function when Z is equal to zero. Finally, just to introduce one more piece of notation, sometimes instead of writing this thing, the shorthand for the derivative is G prime of z. So g prime of Z. In calculus, the. The little dash on top is called prime. So G prime of Z is a shorthand for, in calculus for the derivative of the function of g with respect to the input variable z. And then in a neural network we have A equals G of Z right equals this, then this formula also simplifies to A times one minus a. So sometimes the implementation you might see something like G prime of Z equals a times one minus a. And that just refers to, you know, the observation that G prime, which just means the derivative is equal to this over here. And the advantage of this formula is that if you've already computed this value for a, then by using this expression you can very quickly compute the value for the slope for G prime. Alright, so that was the sigmoid activation function. Let's now look at the tanh activation function. Similar to what we had previously, the definition of D DZ G of Z is the slope of G of z at a particular point of Z. And if you look at the formula for the hyperbolic tangent function, and if you know calculus, you can take derivatives and show that this simplifies to this formula. And using the shorthand we had previously, we're going to call this G prime of Z again. So if you want, you can sanity check that this formula makes sense. So for example, if z is equal to 10, tanh of z will be very close to 1, this goes from plus 1 to minus 1. And then g prime of Z according to this formula will be about one minus one squared. So derivative close to zero. So that was if Z is very large, the slope is close to zero. Conversely, if Z is very small, say Z is equal to minus 10, then tanh of z will be close to minus 1. And so g prime of z will be close to 1 minus negative 1 squared, so it's close to 1 minus 1, which is also close to 0. And then finally, if z equal to 0, then tanh of z is equal to 0. And then the slope is actually equal to 1, which is which is actually the slope when Z is equal to zero. So just to summarize, if A is equal to G of z, so if A is equal to this tanh of z, then the derivative G prime of Z is equal to one minus A squared. So once again, if you've already computed the value of A, you can use this formula to very quickly compute the derivative as well. Finally, here's how you compute the derivatives for the relu and leaky relu activation functions. For the relu g of Z is equal to max of 0 comma z. So the derivative is equal to turns out to be 0 if z is less than 0 and 1 if z is greater than 0 and is actually undefined, technically undefined if Z is equal to exactly zero. But. But if you're implementing this in software, it might not be 100% mathematically correct, but it'll work just fine if you. If Z is exactly root of zero, if you set the derivative to be equal to one, or if you set it to be zero, it kind of doesn't matter if you have an exponent optimization. Technically, G prime then becomes what's called a sub gradient of the activation function G, which is why gradient descent still works. But you can think of it as that the chance of Z being 0 point exactly 000000 is so small that it almost doesn't matter what you set the derivative to be equal to when Z is equal to zero. So in practice, this is what people implement for the derivative of Z. And finally, if you are training a neural network with the leaky relu activation function, then G of Z is going to be max of, say, 0.01 z comma z. And so G prime of Z is equal to 0.01 if z is less than 0, and 1 if z is greater than 0. And once again, the gradient is technically not defined when Z is exactly equal to zero. But if you implement a piece of code that sets the derivative or that says G Prime to either 0.01 or to 1, either way, it doesn't really matter when Z is exactly zero, your code will work just so onto these formulas. You should either compute the slopes or the derivatives of your activation functions. Now that you have this building block, you're ready to see how to implement gradient descent for your neural network. Let's go on to the next video to see that.
