Andrew Ng shows how to compute forward propagation for all m training examples at once by stacking them into matrices. Instead of looping through each example, you compute Z1 = W1 X + b1, A1 = sigmoid(Z1), Z2 = W2 A1 + b2, A2 = sigmoid(Z2) in one operation. Columns represent examples, rows represent units.
In the last video, you saw how to compute the prediction on a neural network given a single training example. In this video, you see how to vectorize across multiple training examples and the outcome will be quite similar to what you saw for logistic regression, whereby stacking up different training examples in different columns of a matrix, you'll be able to take the equations you had from the previous video and with very little modification, you change them to make the neural network compute the outputs on all the examples pretty much all at the same time. So let's see the details of how to do that. These were the four equations we had from the previous video of how you compute Z1, A1, Z2 and A2. And they tell you how given an input feature vector X, you can use them to generate a two equals y hat for a single training example. Now, if you have m training examples, you need to repeat this process for say the first training example, x superscript round brackets 1 to compute y hat 1. That's the prediction on your first training example. Then x 2, use that to generate prediction y hat 2, and so on down to xm to generate a prediction Y hat m. And so in order to write this with the activation function notation as well, I'm going to write this as a two square bracket round bracket one and this is a two two and a two m. So this notation, a square bracket two round bracket I. The round bracket I refers to training example I and the square bracket two refers to layer two. Okay, so that's how the square bracket and the round bracket indices work. And so this suggests that if you have an unvectorized implementation and want to compute the predictions for all your training examples, you need to do for I equals 1 to m, then basically implement these four equations. Right? You need I guess, z1i equals w1xi plus b1, a1i equals sigmoid of z1i z2i equals w2 a1i plus b2 and a2i equals sigmoid of z2i. Right, so it's basically you know, these four equations on top, but adding the superscript round bracket I to all the variables that depend on the training example. So adding the superscript round bracket I to x, z and A. If you want to compute all the outputs on your M training samples, what we'd like to do is vectorize this whole computation so as to get rid of this for loop. And by the way, in case it seems like I'm getting a lot of nitty gritty linear algebra, it turns out that being Able to implement this correctly is important in the deep learning era. And we actually chose the notation very carefully for this class to make these vectorization steps as easy as possible. So I hope that going through this nitty gritty will actually help you to more quickly get correct implementations of these algorithms working. All right, so let me just copy this whole block of code to the next slide and then we'll see how to vectorize this. So here's what we had from the previous slide with a for loop going over all m training examples. So recall that we defined the matrix X to be equal to our training examples stacked up in these columns like so. So take the training examples, stack them in columns, so this becomes a N or maybe nx by m dimensional matrix. I'm just going to give away the punchline and tell you what you need to implement in order to have a vectorized implementation of this for loop. Turns out what you need to do is compute capital z1 equals w1 x plus b1, capital a1 equals sigmoid of z1, then capital z2 equals w2 times a1 plus b2, and then a2 equals sigmoid of z2. So if you want. The analogy is that we went from lowercase vector x's to this capital case x matrix by stacking up the lowercase x's in different columns. If you do the same thing for the z's. So for example, if you take z1,1, z1 2, and so on, and these are all column vectors up to Z1 and right? So that's this first quantity, but all m of them and stack them in columns, then this gives you the matrix z1. And similarly, if you look at, say, this quantity, you can take a 11, a 12, and so on and a 1M and stack them up in columns. Then this just as we went from lowercase x to capital case x and lowercase z to capital case z, this goes from the lowercase a, which are vectors, to this capital A1 that's over there. And similarly for Z2 and A2, right? They're also obtained by taking these vectors and stacking them horizontally and taking these vectors and stacking them horizontally in order to get Z capital Z2 and capital E2. One other property of this notation that might help you to think about it is that these matrices say z and A horizontally. We're going to index across training samples. So that's why the horizontal index, you know, corresponds to different training examples. And you sweep from left to right, you're scanning through the training set and vertically, this vertical index corresponds to different nodes in the neural network. So, for example, this node, this value at the top most, top left most corner of the matrix corresponds to the activation of the first hidden unit on the first training example. Um, one value down corresponds to the activation in the second hidden unit on the first training example, then the third hidden unit on the first training example, and so on. So as you scan down, this is you're indexing into the hidden unit number, whereas if you move horizontally, then you're going from the first hidden unit in the first training example to now the first hidden unit in the second training example, the third training example, and so on, until this node here corresponds to the activation of the first hidden unit in the final training example in the m th training example. Okay, so the horizontal lead, the matrix A goes over different training examples, and vertically the different indices in the matrix A corresponds to different hidden units. And a similar intuition holds true for the matrix Z as well, as well as for X, where horizontally it corresponds to different training examples and vertically it corresponds to different features, different input features, which are really different nodes in the input layer of the neural network. So with these equations, you now know how to implement a neural Network with vectorization, I.e. vectorization across multiple examples. In the next video, I want to show you a bit more justification about why the this is a correct implementation of this type of vectorization. It turns out the justification will be similar to what you have seen for logistic regression. Let's go on to the next video.
