C4W4L10 Style Cost Function

In the last video you saw, how to define the content cost function for neural style transfer. Next, let's take a look at the style cost function. So what does the style of an image mean? Let's say you have an input image like this. You're used to seeing a confnet like that, compute features at various different hidden layers. And let's say you've chosen some layer L, maybe that layer, to define the measure of the style of an image. What we're going to do is define the style as the correlation between activations across different channels in this layer L activation. So here's what I mean by that. Let's say you take that layer L activation. So this is going to be a NH by NW by NC block of activations, and we're going to ask how correlated are the activations across different channels? So to explain what I mean by this maybe slightly cryptic phrase, let's take this block of activations and let me shade the different channels via different colors. So in this little example, we have say, five channels, and which is why I have five shades of color here. In practice, of course, in your network we usually have a lot more channels than five, but using just five makes the drawing easier. But to capture the style of an image, what you're going to do is the following. Let's look at the first two channels. Let's look at the red channel and the yellow channel and say how correlated are activations in these first two channels? So, for example, in the lower right hand corner you have some activation in the first channel and, and some activation in the second channel. So that gives you a pair of numbers. And what you do is look at different positions across this block of activations and just look at those two pairs of numbers, one in the first channel, the red channel, one in the yellow channel, the second channel. And you just look at these two pairs of numbers and see when you look across all of these positions, all of these NH by NW positions, how correlated are these two numbers? So why does this capture style? Let's look at an example. Here's one of the visualizations from the earlier video. This comes from again, the paper by Matthew Ziegler and Rob Fergus that I had referenced earlier. And let's say for the sake of argument that the red neuron corresponds to, and let's say for the sake of argument that the red channel corresponds to this neuron, says trying to figure out if there's this, you know, little vertical texture in a particular position in the image. And let's say that this second channel, this yellow second channel corresponds to this neuron which is, you know, vaguely looking for orange colored patches. So what does it mean for these two channels to be highly correlated? Well, if they're highly correlated, what that means is whenever part of the image has this type of subtle vertical texture, that part of the image will probably have this orangeish tint. And what does it mean for them to be uncorrelated? Well, it means that whenever there is this vertical texture, it probably won't have that orangeish tint. And so the correlation tells you which of these high level texture components tend to occur or not occur together in part of an image. And it's the degree of correlation that gives you one way of measuring how often these different high level features, such as vertical texture or this orange tint or other things as well how often they occur and how often they occur together and don't occur together in different parts of an image. And so if we use the degree of correlation between channels as a measure of the style style, then what you can do is measure the degree to which in your generated image this first channel is correlated or uncorrelated with the second channel. And that will tell you in the generated image how often this type of vertical texture occurs or doesn't occur with this orangish tint. And this gives you a measure of how similar is the style of the generated image to the style of the input style style image. So let's now formalize this intuition. For each of the two images, the style image and the generated image, you're going to compute a style matrix. So more formally, let's say that you're using layer L to measure the style. Let's let a subscript ijk be the activation at position IJK in that hidden layer L. So, so this indexes into the position height, width and this indexes into the different channels. So what you're going to do is compute a style matrix for layer L and for the style image. And this will be a NC by NC dimensional matrix. And you do the same thing for the generated image as well. But now let's define this style image. So g define using layer L and on the style image is going to be a matrix where the height and width of this matrix is the number of channels by number of channels. So in this matrix, the k k prime element is going to measure how correlated are channels k and k prime. So more formally, let me define this as sum over I, sum over j of deactivation at position IJ of channel k times Deactivation at the same position IJ but at channel K prime and just multiply these two things together. So I and J sum over the height and width right at that layer L. So you're summing over the different positions, the X and Y positions, corresponding height and width, and then just multiplying out the activations at channel K with channel K prime. And so far I've been using the term correlation. Technically this is the unnormalized cross covariance because we're not subtracting the mean and you're just multiplying out these things. So this is going to be the style matrix for the input style image S, and then you do the same thing for the generated image. So to this is going to be really the same thing. Sum From j equals 1 equals NWL of the same thing ijk a subscript ij. A subscript IJ K prime L. And if you want to distinguish these activations, I guess you could put a superscript round bracket S and G just to distinguish that these are the activations on the style image S and on the generator image G. And we denote these matrices using the capital Alphabet G. Because in linear algebra this is sometimes also called the gram matrix. But I'm just going to call this the style matrix in this video. Oh, sorry, just forgot KK prime there. Right, so these are the formulas for defining the kk prime element of this NC by NC square matrix. So what you're going to do is given an image, compute something called a style matrix, which will measure all those correlations we talked about on the last slide. So more formally, let's let a SuperScript L subscript IJK denote the activation at position IJK in hidden layer L. So I indexes into the height, J indexes into the width, and K indexes across the different channels. So in the previous slide we had five channels, but K will index across those five channels channels. So what the style matrix will do is you're going to compute a matrix called this G superscript around square bracket L. This is going to be an NC by NC dimensional matrix. So it'll be a square matrix. Remember you have NC channels and so you have an NC by NC dimensional matrix in order to measure how correlated each pair of them is. So, so in particular, glk K prime will measure how correlated are the activations in channel K compared to deactivations in channel K prime, where here K and K prime will range from 1 through NC, the number of channels there are in that layer. So more formally, the way you compute G, L and I'm just going to write down the formula for computing one element. So the KK prime element of this, this is going to be sum over I, sum over J of the activation in that layer, IJK times deactivation at ijk prime. So here, remember I and J index across the different positions in the block, indexes over the height and width. So I is the sum from 1 to NH and J is the sum from 1 to NW and K here and K prime index over the channel. So K and K prime range from one to the total number of channels in that layer of the neural network. So all this is doing is, is summing over the different positions of the image over the height and width and just multiplying the activations together of the channels K and K prime. And that's the definition of G K K prime. And you do this for every value of K and K prime to compute this matrix G, also called the style matrix. And so notice that if both of these activations tend to be large together, then GK will be large, whereas if they are uncorrelated, then G KKK prime might be small. And technically I've been using the term correlation to convey intuition. But this is actually the unnormalized cross covariance because we're not subtracting out the mean and this is just multiplying out the these elements directly. So this is how you compute the style of an image. And you'd actually do this for both the style image S and for the generated image G. So just to distinguish that this is the style image, you know, maybe let me add a round bracket S there just to denote that this is the style image for the image size and those are the activations on the image S. And what you do is then compute the same thing for the generated image. So it's really the same thing. Sum of I, sum of J, A I, J, K L A I, J, K, L and the summation indices are the same, right? As follows. And you want to just denote this is the generated image. I'll just put the round brackets G there. So now you have two matrices that capture what is the style of the image S and what is the style of the image G. And by the way, we've been using the Alphabet capital G to denote these matrices in linear algebra. These are also called the gram matrix or these are called gram matrices. But in this video I'm just going to use the term style matrix. But this is term gram matrix that motivates using capital G To denote these matrices. Finally, the cost function, the style cost function. If you're doing this on layer l between S and G, you can now define that to be just the. You can now define that to be just the difference between these two matrices, GLG squared and these are matrices. So I'll just take the Frobenius norm. So this is just the sum of squares of the element wise difference between these two matrices. And just to write this out, this is going to be sum over k, sum over k prime of these differences, skk prime minus glg, kk prime and then the sum of squares of elements. The authors actually use this particular normalization constant 2 times NH NW at that layer, NC at that layer, and then square this, you can put this up here as well. But the normalization constant doesn't matter that much because this cost is multiplied by some hyperparameter B. Anyway, so just to finish up, this is the style cost function of defined using layer L. And as you saw on the previous slide, this is basically the Frobenius norm between the two star matrices computed on the image S and on the image G Frobenius norm squared. And then with an additional normalization constant which isn't that important. And finally, it turns out that you get more visually pleasing results if you use the style cost function from multiple different layers. So the overall style cost function you can define as sum over all the different layers of the style cost function for that layer, which we defined above, weighted by some set of parameters by some set of additional hyperparameters, which you want to note as lambda L here. So what this does is it allows you to use different layers in the neural network, both the early ones which measure relatively simpler low level features like edges, as well as some later layers which measure high level features and cause the neural network to take both low level and high level correlations into account. When computing style and in the programming exercise, you gain more intuition about what might be reasonable choices for this hyperparameter lambda as well. And so just to wrap this up, you can now define the overall cost function as alpha times the content cost between C and g plus beta times the style cost between S and G, and then use gradient descent or a more sophisticated optimization algorithm if you want, in order to try to find an image G that tries to minimize this cost function j of G. And if you do that, you can generate pretty good looking neural artistic. And if you do that, you'll be able to generate some pretty nice novel artwork. So that's it for neural style transfer and I hope you have fun implementing it in this week's programming exercise. Before wrapping up this week, there's just one last thing I want to share with you, which is how to do convolutions over 1D or 3D data rather than over only 2D images. Let's go on to the last video.