Softmax Regression (C2W3L08)

So far, the classification examples we talked about have used binary classification where you had two possible labels, 0 or 1. Is it a cat or is it not a cat? What if you have multiple possible classes? There's a generalization of logistic regression called softmax regression that lets you make predictions where you're trying to recognize one of C or one of multiple clauses rather than just recognize two classes. Let's take a look. Let's say that instead of just recognizing cats, you want to recognize cats, dogs and baby chicks. So I'm going to call cats class one, dogs class two, baby chicks class three. And if none of the above, then there's an other or none of the above class, which I'm going to call Class 0. So here's an example of the images and the classes they belong to. That's a picture of a baby chick. So the class is three. Cat is class one, a dog is class two. That say, I guess that's a koala. So that's a none of the above. So that's class zero, class three, and so on. So the notation we're going to use is I'm going to use capital C to denote the number of classes you're trying to categorize your inputs into. And in this case you have four possible classes, including the other or the none of the above classes class. So when you have four classes, the numbers indexing your classes would be 0 through C minus 1 through capital C minus 1. So in other words, that'd be 0, 1, 2 or 3. In this case, we're going to build a neural network where the output layer has four, or in this case the variable capital Alphabet C output units. So N, the number of units in the output layer, which is layer L, is going to be equal to four, or more generally, it's going to be equal to C. And what we want is for the number of units in the output layer to tell us what is the probability of each of these four classes. So the first node here is supposed to output or we wanted to output the probability that is the other class. Given the input X, this will output probability. There's a cat given X. This will output probability that there's a dog given X that will output the probability. I'm just going to abbreviate babycake to be bc. So probability of a babycake, abbreviated bc given the input X. So here the output labels Y HAT is going to be a four by one dimensional vector because it now has to output four numbers giving you these four probabilities. And because Probabilities should sum to one. The four numbers in the output y hat, they should sum to one. The standard model for getting a neural network to do this uses what's called a softmax layer in the output layer. In order to generate these outputs, let me write down the map and then come back and get some intuition about what the softmax layer is doing. So in the final layer of the neural network, you are going to compute as usual the linear part of the layer. So Z capital L, that's the Z variable for the final layer. So remember, this is layer capital L. So as usual, you compute that as wl times the activation of the previous layer plus the biases for that final layer. Now, having computed Z, you now need to apply what's called the softmax activation function. So the activation function is a bit unusual for the softmax layer, but this is what it does. First we're going to compute a temporary variable which I'm going to call T, which is E to the z l. So this is applied element wise. So zl here is in our example, Zl is going to be four by one. It's a four dimensional vector. So T itself E to the zl that's an element wise exponentiation. T will also be a 4 by 1 dimensional vector. Then the output al is going to be basically the vector T but normalized to sum to one. So al is going to be E to the z l divided by sum From J equals 1 through 4 because we have four classes of T subscript I. So another way of saying this is that al is also a four by one vector and the I th element of this four dimensional vector, let's write that al subscript I. This is going to be equal to ti over over sum of ti. Okay, in case this math isn't clear, we'll do an example in a minute that'll make this clearer. So in case this map isn't clear, let's go through a specific example that will make this clearer. Let's say that you've computed zl and zl is a four dimensional vector. Let's say it's five. What we're going to do is use this element wise exponentiation to compute this vector T. So T is going to be E to the five, E to the two, E to the negative one, E to the three. And if you plug that into calculator, these are the values you get. E to the 5 is 148.4, e squared is about 7.4, e to the negative one is 0.4 and e cubed is 20.1. And so the way we go from the vector T to the vector Al is to just normalize these entries to sum to one. So if you sum up the elements of T, if you just add up those four numbers, you get 176.3. So finally, Al is just going to be this vector T as a Vector divided by 176.3. So, for example, this first node here, this will output e to the five divided by 176.3 and that turns out to be 0.842. So saying that for this image, if this is the value of z, you get the chance of it being cos 0 is 81. And then the next node outputs e squared over 176.3. That turns out to be 0.042. So it's 4.2% chance the next one is e to the negative one over that, which is 0.002. And the final one is E cubed over that, which is 0.114. So about an 11.4% chance that this is class number three, which I guess is the baby chick class, right? So there's a chance of it being class 0, class 1, class 2, class 3. So the output of the neural network al, this is also y hat, this is a four by one vector, where the elements of this four by one vector are going to be these four numbers that we just computed. So this algorithm takes the vector ZL and maps it to four probabilities that sum to one. And if we summarize what we just did to map from ZL to Al this whole computation, computing the exponentiation to get this temporary variable T and then normalizing, we can summarize this into a softmax activation function and say AL equals the activation function G applied to the vector zl. The unusual thing about this particular activation function is that this activation function G, it takes as input a 4 by 1 vector and it outputs a 4 by 1 vector. So previously our activation functions used to take on a single real value input. So, for example, the sigmoid and the RELU activation functions input a real number and output a real number. The unusual thing about the softmax activation function is because it needs to normalize across the different possible outputs. It needs to take in a vector of inputs and then outputs a vector. So one of the things that a softmax class bar can represent, I'm going to show you some examples where you have inputs x1, x2, and these feed directly to a softmax layer that has three or four or more output nodes that then outputs y hat. So I'm going to show you a neural network with no hidden layer, and all it does is compute z1 equals w1 times the input x plus b, and then the output a1 or y hat is just the softmax activation function applied to Z1. So in this neural network with no hidden layer, it should give you a sense of the types of things a softmax function can represent. So here's one example with just raw inputs x1 and x2, a softmax layer with C equals 3 output classes can represent this type of decision boundary. Notice it's kind of several linear decision boundaries, but this allows it to separate out the data into three classes. And in this diagram, what we did was we actually took the training set that's kind of shown in this figure and trained the softMax classifier with three output labels on the data. And then the color on this plot shows fetch holding the outputs of the softmax classifier and coloring in the input based on which one of the three outputs had the highest probability. So we can maybe kind of see that this is like a generalization of logistic regression with sort of linear decision boundaries, but with more than two classes, with instead of the classes being just 0, 1, the class can be 0, 1 or 2. Here's another example of decision boundary that a Sarpex class bar can represent when trained on a data set with three classes. And here's another one, right? So this is but one intuition is that the decision boundary between any two classes will be linear. That's why you see, for example, the decision boundary between the yellow and the red clauses. That's sort of a linear boundary between purple and red is not a linear boundary between the purple and yellow is another linear decision boundary. But, you know, is able to use these different linear functions in order to separate the space into three classes. Let's look at some examples with more classes. So this example with C equals 4. So we've had the green class in softmax can continue to represent these types of linear decision boundaries between multiple classes. So here's one more example with C equals 5 classes. And here's one last example with C equals 6. So this shows the type of things a softmax classifier can do when there is no hidden layer. Of course, you have a much deeper neural network with X and then, you know, some hidden units and then more hidden units and so on. Then you could learn even more complex nonlinear decision boundaries to separate out multiple different classes. So I hope this gives you a sense of what a softmax layer or the softmax activation function in a neural network can do. In the next video, let's take a look at how you can train a neural network that uses a softmax layer.