Andrew Ng explains why neural networks need nonlinear activation functions: composing linear (identity) activations across layers collapses the network to a single linear function (so added hidden layers give no extra expressiveness). He walks through a two-layer algebraic example showing A2 = W'X + B' when hidden activations are linear, notes that nonlinearities (ReLU, tanh, sigmoid, leaky ReLU) in hidden units are required to learn complex functions, and mentions that a linear activation may be acceptable only in the output layer for regression problems (or to enforce nonnegativity using ReLU). The lecture closes by previewing computing derivatives of activation functions for gradient descent.
Chapters
Why does your neural network need a nonlinear activation function? Turns out that for your neural network to compute interesting functions, you do need to pick a nonlinear activation function. Let's see why. So here's the four prop equations for the neural network. Why don't we just get rid of this, get rid of the function G and set A1 equals Z1. Alternatively, you could say that G of z is equal to Z, right? Sometimes this is called the linear activation function. Maybe a better name for it would be the identity activation function, because you know, it just outputs whatever was input for the purpose of this. What if A2 was just equal to Z2? It turns out if you do this, then this model is just computing Y or Y hat as a linear function of your input features X. To take the first two equations, if you have that a1 is equal to z1 is equal to w1x plus b, and if then a2 is equal to z2 is equal to w2 a1 plus b, then if you take this definition of a1 and plug it in there, you find that a2 is equal to w2 times w1x plus b1. Let's clean that up a bit. Right? So this is a 1 plus B2. And so this simplifies to W2 W1 X plus W2 B1 plus B2. So this is just. Let's call this W prime B prime. So this is just equal to W prime X plus B prime. If you were to use linear activation functions, or we could also call them identity activation functions, then the neural network is just outputting a linear function of the input. And we'll talk about deep networks later. Neural networks with many, many layers, many, many hidden layers. And it turns out that if you use a linear activation function, or alternatively if you don't have an activation function, then no matter how many layers you your neural network has, all it's doing is just computing a linear activation function. So you might as well not have any hidden layers. Some of the cases that I briefly mentioned, it turns out that if you have a linear activation function here and a sigmoid function here, then this model is no more expressive than standard logistic regression without any hidden layer. So I won't bother to prove that. But you could try to do so if you want. But the take home is that a linear hidden layer is more or less useless because the composition of two linear functions is itself a linear function. So unless you throw a nonlinearity in there, then you're not computing more interesting functions. Even as you go deeper in the Network. There is just one place where you might use a linear activation function. G of Z equals Z, and that's if you are doing machine learning on a regression problem. So if Y is a real number, so for example, if you're trying to predict housing prices, so Y is A is not zero or one, but it's a real number, you know, anywhere from, I don't know, $0 is the price of house up to however expensive right. Houses get. I guess maybe houses can be, you know, potentially millions of dollars. So however much houses cost in your data set. But if Y takes on these real values, then it might be okay to have a linear activation function here so that your output Y hat is also a real number going anywhere from minus infinity to plus infinity. Um, but then the hidden units should not use linear activation functions. They could use relu or tanh or leaky relu or maybe something else. So the one place you might use a linear activation function is usually in the output layer. But other than that, using a linear activation function in a hidden layer, except for some very special circumstances relating to compression that I won't want to talk about, using a linear activation function is extremely rare. Oh, and of course, if you're actually predicting housing prices, as we saw in the Week one video, because housing prices are all non negative, perhaps even then you could use a RELU activation function so that your outputs Y hat are all greater than or equal to zero. So I hope that gives you a sense for why having a nonlinear activation function is a critical part of neural networks. Next, we're going to start to talk about gradient descent and to do that, to set up for our discussion for gradient descent. In the next video I'm going to show you how to estimate or how to compute the slope or the derivatives of individual activation functions. So let's go on to the next video.
