Computing Neural Network Output (C1W3L03)

In the last video, you saw what a single hidden layer neural network looks like. In this video, let's go through the details of exactly how this neural network computes its outputs. What you see is that it's like logistic regression, but repeated a lot of times. Let's take a look. So this is what a two layer neural network looks. Let's go more deeply into exactly what this neural network computes. Now, we've said before that logistic regression, the circle in logistic regression really represents two steps of computation. First, you compute Z as follows. And then second, you compute the activation as a sigmoid function of Z. So a neural network just does this a lot more times. Let's start by focusing on just one of the nodes in the hidden layer. Let's look at the first node in the hidden layer. So I've grayed out the other nodes for now. So similar to logistic regression on the left, this node in the hidden layer does two steps of computation. Right? The first step, and think of as the left half of this node. It computes Z equals W transpose x plus b. And the notation we'll use is these are all quantities associated with the first hidden layer. So that's why we have a bunch of square brackets there. And this is the first node in the hidden layer. So that's why we'll have a subscript one over there. So first it does that and then the second step is it computes a 11 equals sigmoid of 0z11, like so. So for both Z and A, the notational convention is that ali, the l here in superscript square brackets refers to the layer number, and the I subscript here refers to the node in that layer. So the node we'll be looking at is layer 1, that is a hidden layer node 1. So that's why the superscript and subscripts were both 1:1. So that little circle that first node in the neural network represents carrying out these two steps of computation. Now let's look at the second node in the NE of the second node in the hidden layer of the neural network. Similar to the logistic regression unit on the left, this little circle represents two steps of computation. The first step is it computes Z. This is still layer one, but now it's the second node equals w transpose x plus b12 and then a sigmoid of z12. And again, feel free to pause the video if you want, but you can double check that the superscript and subscript notation is consistent with what we had written written here. Above in purple. So we'll talk through the first two hidden units in the neural network. Hidden units 3 and 4 also represent similar computations. So now let me take this pair of equations and this pair of equations and let's copy them to the next slide. So here's our neural network, and here's the first and here's the second equations that we worked out previously for the first and the second hidden units. If you then go through and write out the corresponding equations for the third and fourth hidden units, you get the following. And just make sure this notation is clear. This is the vector W11. This is a vector transpose times X. Okay, so that's what the superscript T there represents. This is a vector transpose. Now, as you might have guessed, if you're actually implementing a neural network, doing this with a for loop seems really inefficient. So what we're going to do is take these four equations and vectorize. So we're going to start by showing how to compute Z as a vector. It turns out you could do it as follows. Let me take these W's and stack them into a matrix. Then you have W11 transpose. So that's a row vector, this is a column vector. Transpose. Gives you a row vector. Then W12 transpose, W13 transpose, W13 transpose. And so this, by stacking those four W vectors together, you end up with a matrix. So another way to think of this is that we have four logistic regression units there, and each of the logistic regression units has a corresponding parameter vector W. And by stacking those four vectors together, you end up with this four by three matrix. So if you then take this matrix and multiply it by your input features, x1, x2, x3, you end up with by how matrix multiplication works, you end up with W1, 1 transpose x, W1 W2 1 transpose x, W3 1 transpose x, W4 1 transpose x. And then let's not forget the Bs. So we now add to this a vector B1, 1, B1 2, B1 3, B1, 4. So that's basically this. Then this gives B11, B1 2, B13, B1, 4. And so you see that each of the four rows of this outcome correspond exactly to each of these four row to each of these four quantities that we had above. So in other words, we've just shown that this thing is therefore equal to z11, z12, z1,3, z1,4, right? As defined here. And maybe not surprisingly, we're going to call this whole thing the vector Z1, which is taken by stacking up these individuals of Z's into a column vector. When we're vectorizing, one of the rules of thumb that might help you navigate this is that when we have different nodes in a layer, we'll stack them vertically. So that's why when you had z11 through z1,4, those corresponded to four different nodes in the hidden layer. And so we stacked these four numbers vertically to form the vector z1. And to use one more piece of notation, this 4 by 3 matrix here, which we obtain by stacking the lowercase, you know, W11, W12 and so on, we're going to call this matrix W1. And similarly this vector we're going to call B superscript one square bracket. And so this is a four by one vector. So now we've computed Z using this vector matrix notation. The last thing we need to do is also compute these values of A. And so probably won't surprise you to see that we're going to define A1 as just stacking together those activation values A11 through A14. So just take these four values and stack them together in a vector called A1. And this is going to be sigmoid of Z1, where this now has to be an implementation of the sigmoid function that takes in the four elements of Z and applies the sigmoid function element wise to it. So just to recap, we figured out that z1 is equal to w1 times the vector x plus the vector b1 and a1 is sigmoid times z1. Let's just copy this to the next slide. And what we see is that for the first layer of the neural network, given an input X, we have that z1 is equal to w1 times x plus b1 and a1 is sigmoid of z1. And the dimensions of this are 4 by 1 equals. This was a 4 by 3 matrix times a 3 by 1 vector plus a 4 by 1 vector b. And this is 4 by 1, same dimension as Z. And remember that we said X is equal to a zero, right? Just like Y hat is also equal to a two. So if you want, you can actually take this x and replace it with a zero, since a zero is if you want as an alias for the vector of input features x. Now through a similar derivation, you can figure out that the representation for the next layer can also be written similarly, where what the output layer does is it has associated with it set the parameters W2 and B2 so W2 in this case is going to be a 1x4 matrix and B2 is just a real number as one by one. And so Z2 is going to be a real number. I'm just write as a one by one matrix matrix. It's going to be a one by four thing times A was four by one plus B two is one by one. And so this gives you just a real number. And if you think of this last output unit as just being analogous to logistic regression, which had parameters W and B, W really plays an analogous role to W2 transpose, or W2 is really W transpose and B is equal to B2. So if we were to, you know, cover up the left of this network and ignore all that for now, then this is just this last output unit is a lot like logistic regression, except that instead of writing the parameters as W and B, we're writing them as W2 and B2 with dimensions one by four and one by one. So just to recap, for logistic regression, to implement the output or to implement a prediction, you would compute Z equals W transpose x plus B and A or Y hat equals A equals sigmoid of Z. When you have a neural network with one hidden layer, what you need to implement to compute this output is just these four equations. And you can think of this as a vectorized implementation of computing the output of first these four logistic regression units in the hidden layer. That's what this does. And then of this logistic regression in the output layer, which is what this does. I hope this description made sense. But the takeaway is to compute the output of this neural network, all you need is those four lines of code. So now you've seen how given a single input feature, vector X, you can, with four lines of code, compute the output of this neural network. Um, similar to what we did for logistic regression, we'll also want to vectorize across multiple training examples. And uh, we'll see that by stacking up training examples in different columns in the matrix with just slight modification to this, you also, similar to what you saw in logistic regression, be able to compute the output of this neural network not just on one example at a time, but on your, say, your entire training set at a time. So let's see the details of that in the next video.