Activation Functions (C1W3L06)

When you build your neural network, one of the choices you get to make is what activation function to use in the hidden layers as well as at the output units of your neural network. So far, we've just been using the sigmoid activation function, but sometimes other choices can work much better. Let's take a look at some of the options in the forward propagation steps for the neural network. We had these two steps where we use the sigmoid function here, so that sigmoid is called an activation function. And here's the, you know, familiar sigmoid function A equals 1 over 1 plus E to the negative z. So in the more general case, we can have a different function G of Z, which can write here where G could be a nonlinear function that may not be the sigmoid function. So, for example, the sigmoid function goes between 0 and 1. An activation function that almost always works better than the sigmoid function is the tanh function or the hyperbolic tangent function. So this is Z, this is a, this is A equals tanha of z, and this goes between plus one and minus one. The formula for the tanh function is E to the z minus E to the negative Z over their sum and is actually mathematically a shifted version of the sigmoid function. So is a, you know, sigmoid function just like that, but shift it so that it now crosses a point and rescales so it goes between minus one and plus one. And it turns out that for hidden units, if you let the function G of z be equal to tanh of z, this almost always works better than the sigmoid function, because with values between plus one and minus one, the mean of the activations that come out of your hidden layer are closer to having a zero mean. And so, just as sometimes when you train a learning algorithm, you might center the data and have your data have zero mean, using a tanh instead of a sigmoid function kind of has the effect of centering your data so that the mean of your data is closer to zero rather than maybe 0.5. And this actually makes learning for the next layer a little bit easier. We'll say more about this in the second course when we talk about optimization algorithms as well. But one takeaway is that I pretty much never use the sigmoid activation function anymore. The tanh function is almost always strictly superior. The one exception is for the output layer, because if y is either 0 or 1, then it makes sense for Y hat to be a number that you want to output that's between 0 and 1 rather than between minus 1 and 1. So the 1 exception where I would use the sigmoid activation function is when you are using binary classification, in which case you might use the sigmoid activation function for the output layer. So G of Z2 here is equal to Sigma of Z2. And so what you see in this example is where you might have a TANH activation function for the hidden layer and sigmoid for the output layer. So deactivation functions can be different for different layers. And sometimes to denote that the diff activation functions are different for different layers, we might use these square brackets superscripts as well to indicate that G of square bracket 1 may be different than G of square bracket 2. Right. And again, square bracket 1 superscript refers to this layer and superscript square bracket 2 refers to the output layer. Now, one of the downsides of both the sigmoid function and the TANH function is that if Z is either very large or very small, then the gradient or the derivative or the slope of this function becomes very small. So if Z is very large or Z is very small, the slope of the function, you know ends up being close to zero. And so this can slow down gradient descent. So one other choice that is very popular in machine learning is what's called the rectified linear unit. So the RELU function looks like this and the formula is A /max of 0/ Z. So the derivative is 1, so long as z is positive and the derivative or the slope is zero when Z is negative. If you're implementing this technically, the derivative when Z is exactly zero is not well defined. But when you implement this in the computer, the odds that you get exactly z equals 0000000000000 is very small, so you don't need to worry about it. In practice, you could pretend the derivative when z is equal to 0, you can pretend it's either 1 or 0 and. And your code will work just fine. So the fact that it's not differentiable, the fact that. So here are some rules of thumb for choosing activation functions. If your output is 0,1 valued, if you are using binary classification, then the sigmoid activation function is a very natural choice for the output layer. And then for all other units, relu or the rectified linear unit is increasingly the default choice of activation function. So if you're not sure what to use for your hidden layer, I would just use the RELU activation function. It's what you see most people using these days, although sometimes People also use the TANH activation function. One disadvantage of the RELU is that the derivative is equal to zero when z is negative. In practice, this works just fine. But there is another version of the RELU called the leaky relu. We'll give you the formula on the next slide. But instead of it being zero when Z is negative, it just takes a slight slope, like so. So this is called the leaky relu. This usually works better than the RELU activation function, although it's just not used as much in practice. Either one should be fine, although if you had to pick one, I usually just use the relu. And the advantage of both the RELU and the leaky RELU is that for a lot of the space of Z, the derivative of the activation function, the slope of the activation function is very different from zero. And so in practice, using the RELU activation function, your neural network will often learn much faster than you're using the TANH or the sigmoid activation function. And the main reason is that there's less of this effect of the slope of the function going to zero, which slows down learning. And I know that for half of the range of Z, the slope for RELU is zero. But in practice, enough of your hidden units will have Z greater than zero. So learning can still be quite fast for most training examples. So let's just quickly recap the pros and cons of different activation functions. Here's the sigmoid activation function. Um, I would say never use this except for the output layer. If you are doing binary classification or maybe almost never use this. Um, and the reason I almost never use this is because the TANH is pretty much strictly superior. So the TANH activation function is this, And then the default, the most commonly used activation function is the relu, which is this. So if you're not sure what else to use, use this one. Um, and maybe, you know, feel free also to try the leaky relu, where, um, might be 0.01 z comma z. Right. So a is the max of 0.01 times z and z. So that gives you this bend in the function. And you might say, you know, why Is that constant 0.01? Well, you can also make that a another parameter of the learning algorithm. And some people say that works even better, but I hardly see people do that. So, but if you feel like trying it in your application, you know, please feel free to do so and you can just see how it works and how, how well it works and stick with it. If it gives you a good result. So I hope that gives you a sense of some of the choices of activation functions you can use in your neural network. One of the themes we'll see in deep learning is that you often have a lot of different choices in how you build your neural network, ranging from number of hidden units to the choice of activation function to how you initialize the weights, which you'll see later. A lot of choices like that. And it turns out that it's sometimes difficult to get good guidelines for exactly what will work best for your problem. So throughout these three courses, I'll keep on giving you a sense of what I see in the industry in terms of what's more or less popular, but for your application. With your application's idiosyncrasies, it's actually very difficult to know in advance exactly what will work best. So a common piece of advice would be if you're not sure which one of these activation functions work best, you know, try them all and, and evaluate on, like, a holdout validation set or like a development set, which we'll talk about later and see which one works better and then go of that. Um, and I think that by testing these different choices for your application, you'd be better at future proofing your neural network architecture against the. The idiosyncrasies of your problem as well as evolutions of the algorithms, rather than, you know, if I were to tell you, always use a relu activation and don't use anything else that, that just may or may not apply for whatever problem you end up working on. You know, either you do in the near future or in the distant future. All right, so that was a choice of activation functions, and you've seen the most popular activation functions. There's one other question that sometimes we get asked, which is, why do you even need to use an activation function at all? Why not just do away with that? So let's talk about that in the next video and where you see why neural networks do need some sort of nonlinear activation function.