CS231n Winter 2016: Lecture 9: Visualization, Deep Dream, Neural Style, Adversarial Examples

So today we'll talk about, uh, understanding and visualizing convolutional neural networks. This is one of my favorite lectures to give because we'll get to look at a lot of pictures, videos, have a lot of fun, and there's a lot of space for kind of intuitive understanding of what ConvNets are doing. Before we dive into it, just some of the administrative items. Um, Assignment 1 is now go— now graded. We'll send out the grades tonight. I'm saying or so because just in case that something goes wrong and we can't find a way to send it out to you, Uh, but de— uh, definitely we'll try to have it by tonight. Assignment 2, just a reminder, is due this Friday. Uh, you have to submit it differently from last time. So last time you submitted on the Dropbox in coursework. This time please submit in an assignment tab. That's a difference from last assignment. Uh, so for assignment 2, submit it in the assignments tab. And as a reminder, midterm is next week on Wednesday. And so I posted some information on the Piazza about the midterm and the sample midterm as well. Oh, and some, uh, small fun piece of news is that ResNets, the winning models from 2015, the large 152-layer model is now, um, has now been uploaded, the weights have been uploaded to, uh, to GitHub. And so you can use these models now in your projects if you like. So this is a Caffe model, but I think several other packages should be able to load it as well, but we're not 100% sure about it, so you have to check it out. Um, but yeah. So we're talking about convolutional networks. We saw that ConvNets are useful in a wide variety of application domains. Uh, in two lectures ago, we saw, uh, how convolutional networks work. So we have these conv, pool, fully connected layers. We're stacking them up into architectures. We looked at all the winning architectures over the last few years, and you got a sense of how we actually wire up, uh, ConvNets to, uh, to get, uh, award-winning performance. Um, and then a lecture ago, Justin was talking about object detection localization. So, um, in this case, we looked at RCNN Fast, RCNN Faster, RCNN, YOLO, and really how you have this idea of multiple heads on top of a ConvNet. So some heads are doing classification, some heads are doing regression, and they're trying to just localize all these things and images in various ways, and we looked into that. So this lecture is mostly about understanding and visualizing. So we'll look at various ways in which we can visualize how ConvNets work. So you run a ConvNet on ImageNet and you see that your performance is, say, 5% top 5 error, and you can tell that it's working very, very well. About exactly how does this performance come about and how can we study it or understand what's happening. So we'll go through several of these. I think we'll spend most of the time on the very last one. So I'll kind of like breeze through some of them, some of the earlier ones. So probably the simplest way to actually understand what ConvNets might be doing is to look at the raw activations of a ConvNet. So in a convolutional network, we pass in the image at the bottom. We get all of these different activation volumes in between. And at the end, we have a classifier on the top. So one of the ways you can study what these ConvNets are doing, for example, is we can pick on an arbitrary neuron, say in a, pool 5 layer, and we can pipe lots of images through ConvNet and just see what excites that neuron the most. So then you get visualizations like this where, for example, every row here corresponds to a neuron, and every single image here is something that act— uh, excites that neuron a lot based on the entire training set of ImageNet. And so you can see that some neurons basically they like spotlights, uh, either on the person's forehead or, uh, just in general. Uh, some neurons are more interpretable than others. So some of them like text, and some of them like dogs or American flags. So it's not clear exactly what's going on. So not all of them as interpretable as, as others, but it kind of tells you something a bit. Another way we can study this is not just looking at the activations, but looking at the weights. And in particular, we saw that we can look at the first convolutional layer in the ConvNet, and these, uh, this first convolutional layer consists of a filter bank of filters that we slide over the image. And so we can visualize the raw filter weights, And we can see that the ConvNet on the very first layer is looking at all of these different rotated gabor-like features, as we like to call them. Now, the problem with this visualization, of course, is that this only makes sense to do on the very first layer where you actually have contact with the raw image. But for the other convolutional layers, they're looking at the convolutional layers below, and so their weights are not very interpretable. So you can still do it. So for example, in my ConvNet.js demo, I still visualize the weights of higher convolutional layers. So in the first layer, you can see conv1 filters, and they make sense. The second one is a conv2 filter bank, and every single array in brackets corresponds to a single kind of filter or kernel. And you can see that there's some structure to them, but it doesn't make too much sense. It's not very easily interpretable because these act over activations on the previous conv volume, not on the raw image. So this doesn't make as much sense, and it's not as nice to look at. Now, you might think that, for example, the fact that this emerged just from training is amazing that you can get these Gabor-like filters out of these networks. It turns out that there's actually It's not actually true in the sense that there's a bit of a— there were many papers that were written before ConvNets where there's many different unsupervised and supervised objectives you can come up with. And pretty much anything you throw at an image where you're trying to learn a feature, with different algorithms you almost always achieve these Gabor-like filters. So it's not super exciting. There's a bit of a fatigue about it. But basically— I just wanted to point that out as kind of like a fun aside. In fact, the converse is true. It's hard to come up with a reasonable algorithm that does not give you Gabor filters on the first layer. I think the only example I'm familiar with, for example, is PCA. It doesn't give you Gabor. It gives you these kind of wave sinusoids and so on. But anyways, um, so another way to study ConvNets is in this more global representation kind of view. So we looked at filters, we looked at the weights. Another way to study this is we can take a lot of images through a ConvNet and we can look at what I refer to as FC7 features or codes or something like that. Um, and there's 4,096 numbers just before the classifier, right? And so we can interpret those 4,096 numbers as summarizing the content of the image. And so we can think of these as codes. And so lots of images, as we put them through, we get 4,096 numbers for each one of them, and we have a large collection in this 4,096-dimensional space. And now one way to visualize this, uh, is, for example, using something called t-SNE visualization. I won't go into details how this works, but it's something that every one of you should be familiar with, aware of, that should be in your toolkit. It's very— it makes very nice pictures. Intuitively, what t-SNE does is you give it a collection of high-dimensional vectors, and it finds an embedding in 2 dimensions such that points that are nearby in the original space are nearby in the embedding. And it does that in a particularly clever way that actually gives you very nice-looking pictures. So here, for example, is the MNIST dataset of digits. Which are 28 by 28 images. So this is, I think, 768-dimensional vectors, and they're embedded in 2D, and you can see that the digits from 0 to 9, they kind of cluster in this visualization. So we can run t-SNE on codes of images in ImageNet, and when you do that, you end up with some nice visualizations. So what I've done here is I take all the codes and I embed them, and whatever is close in this visualization is basically close in the mind of the network. You can sort of look at it that way. So there's lots of embeddings here on this website. I preloaded some of it. So this is in the raw space, and then these are some visualizations where I've actually like warped out the representation to make it a square. And then here's actually if we try to zoom in. So these are all— anything that's nearby is basically very similar looking to the ConvNet. And so you can see all the headlights kind of cluster here, keyboards, all the circular things. Then we go down, all the spaghetti, mushrooms. There's a cluster on the right here that's all the dogs are, all the animals in general. Some people cluster in the middle here, boats, I don't know, so random things basically. They cluster in a very interesting way. And so that gives you an idea about what ConvNet considers to be similar to each other. So you can go through these. They're fun to look at. So that's t-SNE. How about different types of objects? How are they, you know, why are they close to each other? Um, why are they close to each other? It's a, I guess it's a very good question. So a ConvNet has to classify all these different 1,000 categories, right? And it only has a finite capacity, uh, to do that. And so it ends up, um, in its process of trying to find the best ways to fit the data, it just ends up, uh, transforming, transforming your images into some kind of a nice dimensional space where it's easy to classify them. And this is a visualization of that space. So I can't say much more than that, basically. So that's, uh, t-SNE visualizations, fun to look at. Another kind of fun experiment kind of on the side, um, that you might want to maybe perform in your projects, maybe, I'm not sure, uh, is occlusion experiments, which are kind of fun and give you some idea about what ConvNets might be doing. What we'll do here is we'll take an image like the one on the top there, and the ConvNet classifies it as a Pomeranian. It's a specific type of dogs— a dog. And now what we'll do is we'll have a patch of zeros that we're going to slide through the image and occlude part of the image. So there's an occluder here in gray. It's kind of difficult to see. This occluder patch, and we'll slide it spatially through the image. And as we do that, we are looking at the probability of Pomeranian and how it varies as a function of the spatial location of that occluder. And so we're going to carve out and activate a kind of a heat map here of what happens to the probability of Pomeranian as you slide this through. And so what might you expect, for example, from this kind of visualization? What would be nice— a nice result? Yeah, go ahead. [Speaker:AUDIENCE] When you cover up the face of the dog. [Speaker:ANDREJ KARPATHY] Yes, so you'd like— so when you cover up the face of this dog, you'd like maybe the probability to go down. So that's in fact basically what happens. And so what you see is the ConvNet doesn't care if you occlude any of these regions, but when you occlude the face of this dog, suddenly the probability of Pomeranian plummets. And so that gives you some confidence that basically this is the features that really matter in the prediction. It's actually looking at the dog. Uh, if you do the same here for a car wheel, then car wheel is the region where your probability plummets. Uh, for this Afghan hound, if you occlude the dog, then you go down to, uh, the probability goes down by a lot. Interestingly, in this picture, by the way, if you occlude the person's head, then, uh, basically the probability actually goes up. So it's kind of like the convnet is not really sure if there's a class here, like maybe there's some class related to a hat on a person or something like that. But once you occlude that region, the ConvNet becomes more certain that actually Afghan Hound is the class for this image. So that's kind of interesting. Um, okay, so that's kind of on the side, a few things that I kind of marched through as bullet points. We're now going to go into, uh, two more types of visualizations, and I'll actually show you this video. I'll actually play the 4-minute video in a second. Throughout this video, because it's a very good video by Jason Yosinski, um, and he has this toolbox called DeepBizToolbox, and you can download this code, then you can run it. And what it's doing, as you'll see, is it's running ConvNet in real time, and you can sub in your camera feed, and you can play with the COMNET and see all the activations. And throughout the visualization, he'll be showing some— he'll be showing two separate kinds of visualizations for neurons. One is deconv, and the other is optimization-based. And we'll cover those later in the class. But for now, I guess these kinds of pictures, you'll see them. We'll cover them in a bit. Let's look at the video. It's a fun video.

Recent advances—

oh, shoot. I didn't fully think this through. How do I get lots of volume here? I'm not sure even sure if— okay, we have to be very quiet basically.

Recent advances in neural networks have enabled computers to better see and understand the world. They can recognize school buses and zebras and can tell the difference between Maltese Terriers and Yorkshire Terriers. We now know what it takes to train these neural networks well, but we don't know so much about how they're actually computing their final answers. We developed this interactive deep visualization toolbox to shine light into these black boxes, showing what happens inside of neural nets. In the top left corner, we show the input to the network, which can be a still image or video from a webcam. These black squares in the middle show the activations on a single layer of a network, in this case the popular deep neural network called AlexNet running in Caffe. By interacting with the network, we can see what some of the neurons are doing. For example, on this first layer, the unit in the center responds strongly to light to dark edges. Its neighbor, one neuron over, responds to edges in the opposite direction, dark to light. Using optimization, we can synthetically produce images that light up each neuron on this layer to see what each neuron is looking for. We can scroll through every layer in the network to see what it does. Including convolution, pooling, and normalization layers. We can switch back and forth between showing the actual activations and showing images synthesized to produce high activation.

We'll go into this in a bit.

By the time we get to the 5th convolutional layer, the features being computed represent abstract concepts. For example, this neuron seems to respond to faces. We can further investigate activate this neuron by showing a few different types of information. First, we can artificially create optimized images using new regularization techniques that are described in our paper. These synthetic images show that this neuron fires in response to a face and shoulders. We can also plot the images from the training set that activate this neuron the most, as well as pixels from those images most responsible for the high activations, computed via the deconvolution technique. This feature responds to multiple faces in different locations. And by looking at the dconv, we can see that it would respond more strongly if we had even darker eyes and rosier lips. We can also confirm that it cares about the head and shoulders, but ignores the arms and torso. We can even see that it fires to some extent for cat faces. Using backprop or dconv, we can see that this unit depends most strongly on a couple units in the previous layer conv4. And on about a dozen or so, you conv 3. Now let's look at another neuron on this layer. So what's this unit doing? From the top 9 images, we might conclude that it fires for different types of clothing. But examining the synthetic images shows that it may be detecting not clothing per se, but wrinkles. In the live plot, we can see that it's activated by my shirt, and smoothing out half of my shirt causes that half of the activations to it's a decrease. Finally, here's another interesting neuron. This one has learned to look for printed text in a variety of sizes, colors, and fonts. This is pretty cool because we never asked the network to look for wrinkles or text or faces. The only labels we provided were at the very last layer, so the only reason the network learned features like text and faces in the middle was to support final decisions at that last layer. For example, the text detector may provide good evidence that a rectangle is in fact a book seen on edge, and detecting many books next to each other might be a good way of detecting a bookcase, which was one of the categories we trained the net to recognize. In this video, we've shown some of the features of the DeepViz toolbox and a few of the things we've learned by using it. You can download the toolbox at this URL and explore for yourself. If you'd like to share what you find, you can and use the hashtag #deepviz. Thanks for listening, and we look forward to seeing what you discover.

Yep, so that's DeepViz Toolbox. Pretty cool. Did you guys hear it, by the way? Was it— OK, awesome. So that's pretty cool, right? So you can investigate these and debug them in real time. So two kinds of visualizations, again, that he presented there are deconv-based approach and optimization-based approach. And we'll go into those now. one by one. So I'll show you first how we generate some of the deconv images that he showed. Um, so before we actually dive into it, I'd like to kind of start off with a question. Um, suppose I feed an image through a convolutional network and we get all these activations throughout. Think about how can you compute the gradient of any arbitrary neuron in the network with respect to the image. So in other words, the gradient on the image for any arbitrary neuron in the network is computable. Of course, it's a smooth function because the ConvNets are a differentiable function. But how would you actually compute that in a, in a code base? Like, say you're working on your assignment and you have your forward, backward, your layers. What is a practical, easy way to actually compute this gradient with respect to any arbitrary neuron? So normally what we're computing gradient with respect to is we have this entire computational graph where we pass the image through and we get a loss at the end. And then we, uh, if you remember your computational graphs, um, your computational graphs, we, we start off the backpropagation by setting a 1.00 gradient there because we want the gradient with respect to loss with— the gradient of loss with respect to loss is 1.0, and we run backprop backwards and find the influence of all the inputs on that output. So how would you do it now if you wanted a specific unit in the ConvNet and find gradient for it? Backprop from there. Backprop from there. Okay. So we do a forward pass and then suppose we're interested in some layer and that neuron, what would you do? Stop there and backprop. Just stop there and backprop. So you have to start off with like some gradient, right? So what we'll do is we forward until some layer and then we have our activations at that layer and now we're interested in some specific neuron. We'll zero out all the gradients in that layer except for the neuron's gradient, we'll set that gradient to 1.00. And then we run backward from that point on. And then when you backprop to the image, you'll find the gradient of the image with respect to any arbitrary neuron by meddling with the gradients. And so in particular, we want to set the gradient of the one we're interested in to be 1.0. And so, yeah. Why don't we just use like numerical gradient to compute the gradient? Um, why don't we just use the numerical gradient? Because it's just, uh, slower, right? So we could use a numerical gradient to also compute, uh, the gradient with respect to any arbitrary part of the network, but it's just you'd have to iterate over all the pixels one by one, and it would just be slow. So this way we can use the analytic gradient, right? We're just doing backpropagation and just computing it, uh, right away using calculus. So for example, if we wanted gradient not with respect to the loss function, but in this computational graph we wanted the gradient for this guy here, this gate, then what I would do is I would run the inputs forward until this gate, and then I would stop right there, and I would just set 1.00 as the gradient at this point, and I would just run backpropagation from here. And whatever gradients I obtain here are the basically gradients, the influence of these neurons on this guy. So that's how you can compute your influence on any arbitrary part of the network. So when you do that with an image, you— suppose we wanted to, like, pick on an arbitrary neuron. You take an image, you forward it through, and you do what I described. You set the gradients to all 0 except for a neuron you're interested in, and you run backward for them from there. You'll find, for example, looking at an arbitrary neuron, you'll find some— something like this as the gradient on the image. So you can see that it's not very easily interpretable. It's kind of like these blobby things. It looks kind of strange. So deconv approach will change the backward pass a bit to actually give us nicer pictures. So in this case, for example, we'll use what's called guided backpropagation, and we'll actually obtain much cleaner images of, say, this cat's head specifically. So that neuron would be looking— would be positively influenced by this cat's face specifically. And we'll see how we can get rid of these artifacts. And so the way this will work, this deconv approach works, is I'm going to go through a figure from the paper. This is Striving for Simplicity paper. So I have to explain this a bit. So here we have the input image goes through a series of layers. And we get some activation feature map at some point in the ConvNet. And now, as I mentioned, if we'd like to derive the gradient for, for any arbitrary neuron here, say like this neuron in the bottom right, we'd zero out— so these are the gradients now in the second row here. We'd zero out all the gradients except for that neuron, and we do backpropagation to find the reconstructed image, as they call it. But really it's the backward pass here. Now, when we're doing backprop, it turns out that to get your deconv approach to give you nice interpretable pictures, we're going to be running backpropagation just like in a previous slide. But we'll, um, we'll meddle with the backprop for specifically the ReLU layer. Uh, so the ReLU layer just, uh, thresholds things at 0. We'll, we'll change the backward pass for a ReLU layer, and otherwise we'll run backward through the image, and also you'll see in a bit why that kind of makes sense. Um, so we're looking specifically at a ReLU layer here and what it computes during the forward and backward pass during backpropagation. So if we're in some feature— this is some feature map in a ConvNet, and we apply the ReLU operation, which is thresholding all the negative activations to 0. Now, in the backward pass, this ReLU layer finds what the gradient from above is. So this is the gradient going into the ReLU neurons, and the way ReLU works, right, is it just kind of blocks all the gradient for all the parts that had negative activations, right? Because if you're a ReLU neuron and there's a grid of 3 by 3 of them here, if you're a ReLU neuron, then if your input was less than 0, then you act as a switch during the backward pass, right? So if, Gradient comes from here. If you're a ReLU neuron and your input was less than 0, then you block the gradient, and if it was greater than 0, then you let the gradient pass through. So basically, that's what's going on in the backward pass here. So this is just backpropagation right now, and this is how you'd write it, uh, out. So basically, the backward pass for this ReLU neuron is the local gradient, which in this case is just a greater than 0 operation, and then times the gradient from above. So this is the chain rule going on here, how you backpropagate through ReLU. Now, in guided backpropagation, what we'll do instead is we'll change the ReLU layer backward in the following way. We are computing what we had before, so ReLU pass, but in addition, we have this additional term here, which basically is saying that we only backpropagate through our ReLU neurons, only the activations, only the parts, the ReLU neurons that have positive gradient. So, okay, so this is the guided backprop. So normally we'd pass through any gradients that corresponded to ReLU neurons that received less than 0 input. And now for guided backprop, we're additionally going to block out all the gradients that correspond— that are negative gradients. So for example, some of these gradients that ended up passing through this ReLU layer in the backward pass were like -2 or -1 or -1 here. We're actually going to kill those. We only backprop the positive gradients. And so what's going on here is, think about the interpretation for what mean— what it means a positive or negative gradient in this case. We're trying to compute the influence on the input, from the input, on some arbitrary neuron in the ConvNet. And what a negative gradient is saying is that this ReLU neuron has a negative influence on this neuron that we're trying to find what activates it highly. And, uh, and this is ha— this, this piece ha— here has a positive influence and these guys have negative influence and so on. So what we're doing is, as we're recursing this ReLU— modified ReLU neuron throughout the ConvNet, which is made up of this Conv, ReLU, Conv, ReLU blocks, we only end up passing through gradients that have a positive influence, entirely positive influence on the activations without the negative influences. So normally what happens is through the ConvNet, this gradient signal is flowing backwards through many ReLU layers, and there are all these paths of influence between the input to any of the output, uh, neurons that we're trying to compute. and some of the paths are negative paths, and some of them are like positive influences. And this gradient signal is really just neurons communicating to each other what should increase or decrease, and they're kind of fighting each other and adding up and so on. And so what ends up happening with backpropagation is the reason you get weird-looking things is that you have all these paths of influence, some of them positive and some of them negative, from every single pixel to the neuron we're trying to, uh, see what activates it, and they kind of end up canceling out in a weird interference pattern. And so what we're doing in guided backprop is we're only keeping the positive influences and only backpropagating those through, and then you end up with much cleaner images. So what this is saying is that this pixel, there's a chain through the ConvNet of positive, purely positive influences, and there's a very direct, uh, kind of increase, entirely increasing chain of, um, influences through the ConvNet on the neuron that you're trying to compute, intuitively. Uh, go ahead, questions. If the convolution kernel had negative entries, then it's not true, right? Uh, sorry? If the convolution kernel has negative entries, then if we have negative gradient, it might be turned into positive. Yeah. So when you think about a ConvNet, it's a stack of conv, ReLU, conv, ReLU, and so on to a first-order approximation. Here we're only changing the ReLU activations. So in between every ReLU backward, there will be a, there will be a convolutional layer backward, and that contains some positive and negative weights, and so you end up with some positive and negative, gradients on the next ReLU layer. But then we again truncate. We throw off all the negative influences again. And so we keep doing this recursively through. Go ahead. If you did the opposite and only kept the negative gradients, would you get a map of the strongest inhibitory features? Yeah, so the question is, if we only kept the negative gradients instead, do we get a map of negative inhibitory features? I guess so, yes. I've never tried that. It's a good question. Okay. Oh, was there a question? You're basically just applying ReLU backwards. Yes. So we're doing backprop just like everything is normal, just normal backprop, but you have to hack the ReLU backward pass. And in that ReLU backward pass, you don't just do what normally a ReLU neuron wants to do according to backprop, but you make sure that you only backpropagate to positive gradients through ReLU. So could you use different activation function? Like instead of using like a— instead of just Taking positive, taking only very large positive or reasonably large positive values. I see. Like if this was not just a ReLU but say arbitrary neuron, I think, uh, yeah, we'd have the same kind of effect. If it was tanh or something like that, you'd do the same thing. And so what ends up happening when you do this is you just get a much cleaner— you don't have all these positive-negative paths like fighting each other in a way, and you only end up with kind of like these positive influences through the network that ends up getting painted. And so, um, these are the kinds of visualizations you can get for different neurons. Every row here is a neuron, and this ends up, uh, this is I think conv6 and conv9, uh, respectively. And, uh, these are kind of, um, we forwarded some image through, we found that this neuron activates, and then we did this backward pass, but this hacked-up backward pass using the hacked-up ReLU neuron, and these are the kinds of visualizations you, you obtain. And so they're kind of this, um, you can see that some neurons are basically, and this is also showing the raw images, cropped to the part of the image that this neuron is a function of, right? Because these neurons are arranged spatially and their area of influence is different in the spatial image. So these are little crops from images which come from the part of the image that activates this neuron. So you can see that some neurons, these are the raw patches that activate them, that activated them strongly, and this is the kind of deconv pass from here. Now there's another way to do it. Not just, uh, this approach proposed in this paper, but there's also something called a deconvNet. This is from a paper from Matthew Zeiler in 2013, and it, it hacks up in the ReLU backward pass as well, but in kind of a more funny way. It ignores the ReLU gradient and is just passing through the positive gradients, uh, without, without— it doesn't care if the activations coming into the ReLU are positive or negative, it discards that term. And so you can see that backprop, a deconv, and a guided backprop is kind of both of them at the same time. So it's kind of funky stuff going on. But, uh, but for deconvnet, that turns out to actually also work well. And, uh, and then you get also pictures that actually seem to make sense. So in these cases, it's a similar kind of idea to a, um, to the guided backpropagation, but, um, we're not taking into account just like subset of neurons based on what happens to activate in the forward pass. So I don't know if I have very good explanation beyond this point, but basically it produces much cleaner pictures because we're only passing through the positive influences, entirely positive influences through the backward chain. Um, so these are some, uh, visualization for the first layer, for the second layer. Uh, so we can see that some neurons here activate for different kinds of textures. If we go deeper into the network, we start to see that on the third layer, we get neurons that, uh, respond to some kind of like honeycomb shapes, or cars, or yeah, lots of interesting pieces here. And as you go to the fourth layer, we start to get some object-like things. So say dog faces here, and so on. And so we're building up these features one at a time. So every conv is kind of combining these little pieces into a higher layer of abstraction kind of shape that grows over time. Go ahead. [Speaker:AUDIENCE] Why are we considering the gradient? I mean, was there any— intuition behind looking at the gradient? [Speaker:ANDREJ KARPATHY] Yeah, so what this is telling you— the gradient really is telling you— like, the reason that we see, say, red here in this person here, like, that gradient is telling you that if you were to make this person's face redder, there would be more red in it, then that would have a locally positive influence on this neuron's activation. So basically, making a small step in the image in this direction would have positively influenced that neuron downstream, right? That's what the backward pass is telling you. It's telling you— that's what backpropagation is doing, is it's computing the influence, the gradient of every single input to whatever you're interested in, right? So, but in backward pass, you have all these positive, negative influences, and here we're cropping always the negative influences. So you end up with like a subset of the backprop pass almost. You can look at it that way. Okay, so I'm not a huge fan of the deconv visualizations. We're going to go into the optimization-based visualizations, which I think make much more sense and are much more intuitive in terms of what's going on. So let me get into those. What's nice about the deconv approach is that you do a single forward pass, single backward pass, and you're done. You just visualize that hacked-up gradient and you get some visualization that looks pretty. In the optimization approach, we're going to have to do a bit more work computationally, but it makes a little more sense, I think. So what we're going to do here is we're go— we have a ConvNet, and of course everything is fully differentiable here. And what we're going to tru— do now is we're going to try to optimize the image so that we're— so the parameters of our optimization now will be image and we'll be keeping the entire ConvNet fixed. Normally, we optimize over the parameters in ConvNet. Now we'll hold everything fixed. And we're going to be trying to maximize an arbitrary score in the ConvNet. So optimization problem-wise, basically we're trying to find an image i such that your score is maximized and there's some regularization on i. So say L2 regularization, which doesn't, uh, which discourages kind of these pathological cases where some of the pieces of your input might be way too large or something like that. So it's regularization and we're trying to maximize the score. And so the way this will work is we'll start off with say like a zero image. We'll feed it into a ConvNet and get some scores out. And now in terms of the gradient, Again, as before, we're interested in some class score, so we set the gradient at that point to be all zeros except for a one at the score that we're interested in. And then we're going to do backward pass to find what change to make to the zero image. So this is normal backpropagation now. We're not doing any more guided or anything weird with ReLU. This is just straight-up backprop. So forward, and now we are interested in some particular gradient signal because we want to increase some score. We do backward and we do updates. So we're going to iterate this over and over, and we're going to basically run optimization over the image to maximize a class score, uh, subject to also regularization penalty. Okay? So when you do this, the kinds of results you get, and we're doing updates on this image, so we're doing image updates instead of the weight updates, uh, you get images like this. So here we are maximizing any class here, and these are— this is the maximally— this is the image that maximally activates that class's score. So we have dumbbells and cups and dalmatians. You can see all the little structured patterns there. It's kind of fun to look at. Bull pepper, lemon, husky. Here we have— and also notice, for example, here's goose. You can see that the optimization is really trying to take advantage of all the area it possibly has to increase the score of goose. So it, like, tiles the entire input with goose everywhere because it's just, like, trying to blow up the score of goose. So one goose is good, but 10 gooses is much more likely goose. So that's why that happens. Um, one interesting— another kind of interesting, uh, way of interpreting, uh, the gradient signal at the, at the image, uh, and one way you can use it in an interesting way in this paper from, uh, Kieran Stefanian is as follows. Um, they have this— these fun experiments where they take an image like, say, this dog, and they forward the image through the ConvNet, and then they— it says that this is some kind of a special terrier or something. And then you, uh, set the gradient for that terrier to be 1 and you do backprop. And, um, you arrive at your image gradient. And then what they do— this is kind of a confusing notation here, but they, they squish it through channels with a max. So basically they're trying to compute a heat map here, a one-dimensional kind of map of these gradients. And so what might you expect, or what would you hope, or maybe happens in terms of what this might look like? We're visualizing these gradients. [Speaker] High on the object location. [Speaker:ANDREJ KARPATHY] OK, so you'd expect that it's high on the object location. I guess that's reasonable, right? So basically, this is what happens. So you end up with something roughly like that, where the gradients, the max of the gradients without taking into account its sign, are these blobby things. And so really, the way to interpret these maps is its strength of influence of every single pixel on the score. So the fact that some of this stuff is black, what that's telling you is that if you wiggle that pixel, the score of that dog is basically not modified at all. So the ConvNet doesn't care about it. But if you start to meddle with the pixels on the dog, then it's going to actually influence the score in some way of the dog. And so this is kind of telling you area of influence in the image. That gradient signal can be kind of used as a measure of area of influence of every single pixel on the score. Go ahead. So with the slide before, you start at zeros and you arrive at some optimal image. I'm just curious how sensitive is that choice of image? Like if you instead started with random image and optimized that? Uh, yeah. So you're asking how sensitive is, uh, are these optimizations to the initialization? We'll see actual examples of that in like 2 slides. Yeah? Uh, yeah, and also with the previous one, rather than having like that, uh, ridge regression term on there, Why don't we just, like, put a hard max at, like, 255 or something like that?

Yep.

That should be a little bit more of a— Yes, so you're asking about the precise forms of regularization used here and the fact that you could use multiple. There's, in fact, a paper we'll go into in 5 slides where they actually talk about this regularizer quite a bit more. And you want to choose it properly, and there are multiple choices here. So what's interesting here is they have these experiments in the paper where you can actually use that gradient, and you can use a GrabCut segmentation algorithm, which we want I won't go into, but that is the input to GrabCut. And basically GrabCut does this joint optimization over pixels. And you basically want similar pixels or nearby pixels to have the same label. And you can actually crop out some of these images just based on the gradient signal on the raw image. And so I should like to say that these are kind of, I think, I suspect cherry-picked examples in this paper because I think we tried it in some of our stuff. You don't always get this nice-looking things, but in principle, you can imagine that this might work sometimes. Okay. So right now, we were maximizing— I was showing you examples of maximizing the full score, and we were doing optimization over the image. And the point here is that you can actually do this for arbitrary neuron in the ConvNet. We saw this with deconv as well. And so the way this would work is, again, we have an optimization over not just the score, about maybe any arbitrary activation of any neuron in the network. And there's this penalty L2 that we've been using so far. In this paper from Jason, uh, Jason Josinski, they argued that actually there are better ways of regularizing, and they were looking for different ways of regularizing the image to make it more image-like. And they actually found a different kind of more explicit scheme that they found worked better, and we'll see another set of visualizations that make everything look a bit better, I think. And the way they do it is they ignore this penalty. Instead, they just maximize a neuron. But after they do forward and backward, they do an update, and then they blur the image a bit. So if your image here is x, they do forward, backward, they do a small update on the image, and then they blur it. And so this blurring that you keep doing at every single iteration, it's kind of like, uh, preventing the image of accumulating high frequencies, not through the loss function that you're optimizing here, although I think you can re-express it as the blur kernel here corresponding to some total variation loss function. But they do it more explicitly like just a blur, and then they do a bit more parsing encouraging things that I don't want to go into. You can check the details in the paper. But basically end up with cleaner visualizations for classes, I think. So here are some flamingos or pelicans or heartbeast. So these are classes at the very end of the ConvNet, and we're optimizing these images to, to maximize that class. So these images are in fact equivalent to the image that I've shown you before. The experiment is doing the exact same thing as here, but it's using a different regularizer. And so you see that instead of the L2 norm, this one I think looks a bit better. To your question in terms of how much does initialization matter, they are showing 4 different crops here, 4 different results for different initializations. So you can get a sense of like there is some influence of initialization. Okay. So this is at the last layer of scores, but we can actually go down the ConvNet and repeat the optimization process for arbitrary neurons. So on layer 7 and 8, things are not as clear yet. But we— once you descend down the ConvNet, you start to get really interesting visualizations. So on layer 6, it's still kind of a bit of a mess, but I think on layer 5 is where we start to get some very interesting, like, neat-looking things. So this is some little pieces of ocean perhaps, or this is some kind of spider-looking thing or text-looking thing, right? So you can see that these are the images that maximally activate these neurons. They're actually quite interpretable in some cases. On the layer below, we have these pearl-looking things or squiggly-looking things. And so these are— these just come out of the optimization. So this kind of gives us some idea of what these neurons really like to see and what they're firing for, what they're looking for in the image.

Go ahead.

Does the granularity of the feature depend on the total number of layers? Like, would layer 7 be more abstract if you had 200 layers? Yes, so the question is, would the layer 7 be more abstract if you had 200 instead of just like 7 layers? A very difficult question. Um, actually, yeah, it's not clear to me. So one thing that does tend to matter when you do these visualizations is the effective receptive field of different kernels. So if you think about like which parts of the input image could possibly affect different neurons, as you go deeper, there is sort of larger space in the input that actually even has a chance to affect that thing. So that's part of the reason why you tend to get bigger, nicer looking things as you go deeper in the network and visualize these things. Space? Yeah, so think about what we call the effective receptive field of every single neuron along the network. In the first layer, the effective receptive field, say, for a VGG net is like 3 by 3, right? But then as you go deeper in the network, you find that these neurons are a function of larger and larger areas in the original image. And so that's why you see for this layer 4, you see more localized smaller pieces of image. And for higher up, you see kind of more blobby things. So my guess would be if you have 200 layers, by layer maybe 20 maybe like 7 or 8 or 9. You'd have to do the math. But by that layer, you're probably seeing neurons that are a function of the entire image. And so you'd get just these kinds of things, I assume. And then for layer 3, 2, 1, we have again— yeah, it's self-explanatory. Looks fun. [Speaker:AUDIENCE] So that's derivative— I mean, that's for the score, like to optimize the score of a particular class? That's right. So this is achieved by optimization. So these are the images that maximally activate some unit in the neural net. In the, in the last, like, score. Yeah. Uh, not, not on the last, uh, not on the last, uh, layer for score, for any arbitrary in-between, uh, layer of a convnet. That's right. And so this is achieved by optimization. So forward-backward, uh, update image and do this many times and these things just come out. Whereas before we were doing just a single backward in the deconv approach. Okay, another kinds of fun questions we can look into is— where are we? Another kinds of fun questions we can look into is the following question. How much information is there basically in these codes? So when we forward the convolutional network through, we get some code here, the FC7 features, just before the classifier, and that's 4,096 numbers that form a summary of the image. And the question is, can we actually invert the image just given the code? Okay, so you might imagine this might actually have some potential repercussions for privacy where applications may be, because maybe some companies might try to only store the features, not the raw images. And the question is, like, can you actually invert that? And so the way this optimization would look like is we're given a particular feature, and we'll formulate it as just we want to minimize— we want to find an image that best matches that feature, say just as a regression. So we want to match that code and subject to a regularizer on the image. And so instead of maximizing any arbitrary feature, we just want to have a specific feature, and we want to exactly match it in every single dimension. And so when you run this optimization on the image, you get something that looks like this. This is the original image. So we forwarded it through a ConvNet, and we got a code. And then here we are trying to invert it from just that code, just from those 4,000 numbers. And so you can see that some of the structure— this is from different initializations, I believe— Some of those structures are still visible, and so that's roughly gives you a hint of just how much information is present in there. You can actually not do the reconstruction at just the final code layer. You can do it anywhere in the ConvNet. So for example, on the pool-5 layer, which is slightly lower down the network, the pool-5 layer has much more spatial information than the final code layer, um, and so you can actually do a bit better job of reconstructing. So in this case, for example, the position of that bird is actually relatively recoverable just from the code. Um, so quite a lot of information packed in there. Uh, you can also, uh, look at a single image and look at where we, uh, try to recover it from. So you can see that in the end there's very abstract recoveries, doesn't work super well. But as you go down, uh, you have many more activations in the ConvNet. And, uh, when you're very close to the image you can actually do a very good job of reconstructing. So this gives you a sense of exactly how much information is thrown out by the ConvNet as it's doing the forward pass, roughly. Just one more example of this. Say this is a flamingo image. Again, the same thing. As you go lower down the ConvNet, you end up with more and more precise predictions. Okay? So we looked at maximizing activations. We looked at segmentation, kind of. We looked at matching a particular code. So we're now going to go into some other— any questions anymore about Matching code? OK. So I'm going to go into DeepDream. So DeepDream, which I briefly flashed a few lectures ago, you can get really funky images. And as you might imagine by now, this is all done in the same process. We have an optimization process over the image trying to achieve some kind of a loss in the ConvNet. So what is the loss is really the question because the process I think you already have a very good idea about. And so if you go into, um, the GitHub page that Google released with DeepDream, it's basically one IPython notebook, and there's literally, I think, like 100 lines of code in total. And so that's all of DeepDream. It's a very simple process. In fact, I kind of took out a crop here, and this is the core of DeepDream. And so the way this will work, and you can— if you're good at reading this, then you, you can see what DeepDream is doing, but I'm going to go through it, uh, piece by piece. But this is a make_step function here, that you have a current image and make_step will be called repeatedly as we optimize. So every single make_step call does a small update to the image. And, um, notice that the input to make_step is also this variable called end, which is a string denoting exactly what layer we want to dream at. And so basically, we can dream at any layers in the ConvNet. And this is an Inception network, and one of the Inception layers is called Inception, uh, what is that, 4C. So that's where we might want to drink, for example. And the way this will work is, uh, this is some boilerplate stuff here. And look here, we, we're doing net.forward, then we're calling objective, and then we're doing backward and then update. So just to highlight some of the parts here, um, there's a bit of a jittering regularizer here that I don't want you to worry about too much right now. But the important part is this net.forward, what we're doing is we're forwarding the network, then we're calling the objective. On DST, which is the layer at which we want to dream. What's happening there is that DST is a blob in Caffe. This is Python wrapper around Caffe. These blobs have two fields. They have a diff field and a data field. The data field holds the raw activations and the diff field holds the gradients. So, you can see that what the objective is doing is we forwarded the network up to some point and now we're setting the gradients to equal the activations, and then we're doing backward from that point back to the image, and then we're scaling the gradient on the image by some— we're normalizing the gradient and doing an update. And, uh, and this is just clipping and stuff like that. So really what DeepDream is doing here is, if you read into what's happening, is you have a ConvNet, you pass the image through to some layer where you want to dream, and so you pass it through until then, and then the gradients at that point become exactly identical to the activations at that point, and then you backprop to the image. Okay, so intuitively in English, what is that doing? And this is always, you always dream, by the way, I should say, after ReLU units. So these activations are always zero-cropped. So these are ReLU activations. ReLU activations. So what is DeepDream doing? Good. [Speaker] You're kind of amplifying the features —maximally activated the network. You're amplifying the features that maximally activated in the network. Yeah, OK. I think that's very well said. So if you have a collection of ReLU units in the forward pass, and some of them activated more than others. And now remember, when we're computing the influence of any arbitrary piece of image on any arbitrary unit, we're kind of setting the gradient to be like 1.00. But here we're setting the gradient to be the activations. So basically we're, we're trying to find the image update that would boost any of the axis existing activations after the ReLU unit. So whatever we received as the activations, we're trying to boost all of it at the same time. So this, the way this will work is if you take, for example, you get it to look at, uh, clouds and, uh, you end up with this weird optimization that gives you all these little things and, um, So the clouds, basically, think about the ConvNet looking at the clouds. There might be some, say, a dog-like detector that detects dogs normally. And if that dog detector thinks that this looks a bit like a dog face, just a tiny bit, then there will be some activation there. And then in the gradient signal, we're going to say, okay, we want to boost the activation of that dog feature. And so once we do a backward pass, we are finding on the image, how do we change the image so that that becomes more of a dog? So whatever activated just becomes boosted. And so we keep repeating this over and over again. And so we keep plugging this into DeepDream, and this dog just becomes more dog-like over time. And the ConvNet convinces itself that there's just like dogs or whatever else everywhere because it just keeps refining the image. Whatever activates, it just wants to activate more. Okay, so that's roughly what DeepDream is doing. And then you end up with all kinds of funny shapes like Admiral— this is This is from the Google post. They call them the pig snail, the camel bird, the dog fish. You just get all kinds of funky things. And intuitively, the reason you get dogs and fishes and so on is just this was all trained on ImageNet, so there's a huge amount of dogs and in general, like, animals. So you have lots of dog features. So these ConvNets are always very happy to detect dogs. And then if you show them anything dog-like, they'll just fire, and then they'll get boosted, and then you get dogs everywhere. We can also DeepDream, as I mentioned, on any arbitrary layer of the ConvNet. If you DeepDream slightly lower, then those features are not about dogs or anything like that. They're detecting some weird shapes and patterns, right? So those features will kind of, you know, be detecting those kinds of features, and that's what you'll end up reflected in the pictures. And so, yeah, that's basically how DeepDream works. I should mention that this code snippet is really just about the— the core of it. There's one more code snippet that actually applies this on different octaves of the image, so for different scales, and there's a bit more to it. And this jittering is actually kind of important. And so there's some subtleties to it, but basically that's what DeepDream is doing. And now we're going to look at some funny videos because this is the best part. So— So someone was running DeepDream on video, but they were doing this optical flow smoothing where they were trying to ensure consistency from frame to frame. So I don't know, it's just the most trippy video I'm aware of. Okay. I think you guys— it's a pretty long video, but I think you get the idea. So we're gonna stop it here. Um, I can show you one more here. Uh, this is, uh, yeah, this is DeepDreaming, This is also a video from some movie that I haven't seen, but it's something about drugs, and so it's appropriate. And— [Speaker:MAN] '65, the great San Francisco acid wave. [Speaker:ANDREJ KARPATHY] Acid wave, so— [Speaker:MAN] I recall one night in a place called the Matrix. [Speaker:ANDREJ KARPATHY] And so here they do it frame by frame. I don't think they do some smoothing in this case. And I'll just— Just forward, maybe there are some— [FOREIGN LANGUAGE] OK. So you get the idea. So that's DeepDream. So again, it's just optimization over the image. Go ahead. What if you did the opposite? So instead of amplifying those things, So you want to instead reduce the signals of whatever activated? I have no idea what would happen, but you're very welcome to try it for a project or something like that. Okay, so we're going to another— sorry, go ahead. So what were you saying about the reason for all the dogs? Oh, the dogs, like where they come from? It's just there are so many features in the ConvNet that really care about dogs because there's so many of them in ImageNet data that a large fraction of the ConvNet features just really like dogs and have lots of features about dogs because then you need to differentiate between them in the ImageNet dataset. The problem, so for ImageNet there's 1,000 classes and 200 of those are dogs. Because the source of the image that people love dogs, so they upload. Yeah, I don't know exactly how that came about. The reasoning behind that is that sometimes we care not just about classifying broad categories, but they wanted to study the ability to classify within like very fine-grained categories that are very close to each other. So that's the reasoning why there are so many different types of dogs in ImageNet. Yeah. Yeah, you want to have both like broad categories and also fine-grained recognition in between, and you want to kind of test both. Okay. Uh, the next kind of, uh, so we saw some applications. Okay, go ahead. Yeah. So when you say that you're dreaming from layer X, you're picking a neuron from layer X and then pushing it back, right? Yeah. So when we're dreaming, so ConvNet has all these layers. So when we're dreaming at layer X, we only forward up to x, and then at that point, all the neurons, whichever ones activated, we are, we are going to set the gradient. We are, we will want to boost all of them at the same time. Yeah. So we're going to another fun application of, uh, uh, optimizing over the image with ConvNets, and that is Neural Style that some of you may have also seen. This was a paper that came out over the summer and had some really interesting, amazing results. So basically, you can take an original image and you can paint it in a style of something automatically. So for example, here I took a picture of Gandalf and a Picasso painting, and I can actually optimize to get a Gandalf Picasso picture. So this is Gandalf in a style of Picasso, and this is actually achieved with optimization on the raw image with ConvNets, which is amazing. Here are some more examples. So this is me as a Van Gogh style Starry Night. This is my Picasso Gandalf. These are just randomly grabbed yesterday from from a website that allows you to apply neural style to your own images. It's called deepart.io. And so you can create really amazing-looking pictures. I still— we're going to go into how this works, and I still cannot believe that it works this well, basically. It's something very magical is going on. So let's see how neural style works. It's also a simple idea kind of as like DeepDream. So what we're going to do with neural style is the following. We have two images in the input. We have a content image and a style image. and we want to transfer that style onto the content image, okay? So we have content and style images. What we'll do is we'll pass both of them through a ConvNet. So first, we'll take the content image, and we'll pass it through a ConvNet, and we are going to record all the raw activations in the ConvNet. So for example, in VGGNet, there's a layer that might have this activation size. We're just going to store those raw activations. Very simple so far. And we're going to say that these raw activations basically kind of correspond to the content of the image. That's how we're going to call it. To the— to— we're— these raw activations kind of correspond to the content. Now, to extract the style, what we're going to do is we're going to take the style image, pass it again through the ConvNet, but instead of keeping track of the raw activations, what, uh, these authors of this paper found was that actually a very nice kind of statistic of, uh, the style is not in the raw activations but in their pairwise, um, statistics. So what we're going to do is we're going to form all these outer products of all these basically covariance matrices that are spatially invariant in the ConvNet. So, for example, I have to draw this out because it's kind of confusing. We have a 224 by 224 by 64 volume in the first layer of a ConvNet of VGGNet, say. Okay. So, 64 filter activations at all these different locations. And then there's basically 224 by 224, what I'll call fibers. So these are 64-dimensional fibers of activations. And what we want as a statistic of the style of this image is this 64 by 64 matrix of all the outer products of all the fibers in this volume. So we take all the fibers here, we take outer products, and we sum it all up to get a single 64 by 64 matrix of these basically covariances, uh, uh, or Gram matrices as they're called. Of these activations. Another way to look at it is that we're going to take this volume, we're going to reshape it to be one matrix of 64 by 224 times 224. And so this is a very long matrix here, and we call this matrix V. And then the Gram matrix is just V transpose V, so all the outer products, all the outer products added up. And so it's just keeping track of how often each pair of features fires together as you pipe this through the conv— through the volume. So it's kind of looking at statistics of what features like to fire together across spatially the image. So all the outer products summed up across all spatial locations, okay, the Gram matrix. And we are actually going to construct this not only at the first conv location but at all the other conv volumes throughout. And then what we're doing when we're doing optimization for neural style is notice that this Gram matrix construction is just outer products of all these fibers. That's a perfectly differentiable operation, right? And so our loss here will consist of two terms, a content loss and a style loss. So again, we start off with, say, a random image, or you can initialize at the content image, but we start off the initialization somewhere, and then we're going forward and backward to optimize over that image. And the loss is that we want the content to match the content image, and we want the style statistics to match the style statistics of the style image. And so all the activations throughout should match the content activations, uh, so the activations should be the same, same like L2SENSE, and all the gram matrices should also match. So all the pairwise statistics are from the style and all the actual activations are from the content. And so these two guys are basically fighting it out in how they— how they give rise to the backward pass of this, of this image. And so in practice, I think actually we use only a single content layer. We only use like the 5th layer or something like that of content and we only— and we use many more style layers. So we apply, I think, the style loss on multiple layers and we apply content on only one layer. If you want to play with this, by the way, I forgot to mention that Justin actually has a very good implementation in Torch of neural style. That is very popular, and it's on this URL. So you can see the implementation in Torch and see exactly how this can be implemented. But the rough idea is just as with DeepDream, just as with all the other visualization we've seen before, we're optimizing over the image subject to some loss. In this case, the loss is that the activations at some layer should match the activations here. So we want something that gives you activations of similar to what you had when you passed Gandalf through, but we want these pairwise feature activations to match what you have when you pass Picasso through, and that turns out to create these kinds of images. I don't have a very good explanation of exactly how this happens. I still think it's kind of magic, and so I'm not sure exactly how this can possibly work. So I can't explain it any better than that, except for just like showing you the math. So yeah, that's StyleNet. It's kind of unsatisfying because I think it's confusing. But anyways, any questions about StyleNet, maybe? Go ahead. [Speaker:AUDIENCE] The V transpose V matrix, couldn't we see it as some kind of orthogonal projection on the style instead of the deep neural net? [Speaker:ANDREJ KARPATHY] Could you interpret the V transpose V as some kind of a projection I'm not sure if I'm fully seeing that. It's kind of like we take all these fibers and we stack them as columns, and then when we do this giant outer product of this times its transpose, we end up with a 64 by 64 matrix. And I think it's just a matrix of how often each— the features across this depth like to fire together. And that's all summarized in this matrix, which will be, I think, symmetric. Yeah, so it's a symmetric matrix. And it just stores what neurons like to fire together, and that's what we're optimizing for with style. So in here, when we're actually optimizing over this ConvNet, we'd like to have the same kind of pairwise statistics, but we'd like to have the raw activations of that.

Yeah? [Speaker:AUDIENCE] So what if we only use a style loss?

[Speaker:ANDREJ KARPATHY] What if we only use a style loss? Then you end up with kind of mixed up Picasso painting, I think. [Speaker] So it is like kind of regular translation? [Speaker] Yeah, so you're asking what if we only use this loss here for style? You basically will end up with a permuted Picasso. I've tried this out, and you'll end up with this image. It will look like a Picasso painting, but it will be completely permuted because all the spatial information is not reconstructed exactly well, I think. It's mostly like the style stuff just tells you about the kinds of strokes that you have. —like say, for example, here in the Van Gogh, like, these brush strokes are just perfectly summarized. And the content part kind of gives you the more global layout somehow. So yeah, like a texture is better stored in these pairwise activations is kind of like the point there. Yep? I think your microphone may have turned off. My microphone for the class? Yeah. It was 1 bar, so that could be why. When did this happen? Just now? Yeah, it's turned off. Ugh. OK. So actually, a fun thing about neural style. You can— there's actually not— like, the Gram matrix isn't totally a special thing. Like, what you really want is just some statistic of the activations that is spatially invariant. So the Gram matrix is this, like, one statistic you might imagine That's really easy to implement. Another thing you can do is just take the mean feature activation across space. So instead of a 64 by 64 matrix, you just get a vector of length 64 that tells you the mean activation for each feature map. And this actually does something surprisingly as well. So I'm not— there's this interpretation of the Gram matrix as having this pairwise interaction between features, but I'm not totally sure that that's exactly right. Exactly what is making it work. Yeah, it's a good point. I think— so this is a paper from the summer, so I think a lot of people are still trying to figure out exactly what's going on here. And so Justin, you're pointing out that if you just take a mean over here, the important part is actually the spatial invariant part of it, not maybe— yeah. There's actually another paper that came out on archive maybe 2 weeks ago that uses a Markov random field to measure these interactions rather than than this Gram matrix, and they have some results that look really crazy, but it's like 10 times slower. Um, so I tried it out and it took a long time even on a really good GPU. Yeah. Another point, uh, I'd like to make actually, by the way, I thought of is, uh, this is best optimized with LBFGS because we don't have a huge dataset, and so everything fits in memory, and it's just a nice optimization here problem with a single image that we're backpropping through. So this is a very nice kind of, use case for LBFGS. So the second-order stuff works much better here than, I think, Adam and so on. So this is an example of where LBFGS would be very useful. Okay, so we'll get into the last part here, which is also very interesting and kind of confusing. And so the question is, we saw all these optimizations over the image, we can do lots of arbitrary stuff. So one question you might have is, can we use this optimization process process to full ConvNets? Spoiler alert, we can, and I mean it in the following way. What we'll do, and these are experiments from 2013 now by, um, by Christian Sagati et al. And what they did here is they took an image, say of this school bus, and then we basically— everything is differentiable, so we can ask for the gradient on that image for making, like, for the ostrich class. So basically, ostrich class, We back— so we forward the image, we set all the gradients to be 0 except for ostrich where we set 1, and then we do backward pass. We get a gradient of what to change in the image to make that more ostrich-like. And what you'd hope to see is, as we run this optimization, you'd hope that this bus turns into an ostrich over time, but that is not in fact what happens. What happens is, uh, somewhat depressingly, that it turns out that the distortion you need to actually get a very confident ostrich out of that image is this, and this is a very likely ostrich, but to you and I, it looks just like a bus. So that's kind of confusing. And so you can turn anything into an ostrich. You can turn actually anything— it's not even just about ostrich. You can turn anything into anything, and you need only depressingly small changes. So that's kind of weird. What is going on? So we're going to go over why this happens. You can also not just do it from the random image. You can do it from random noise. So, uh, we can start off with noise and we can ask to change the image such that we get a robin or a cheetah or something with very high confidences. We can create these arbitrary images that basically become the things are of this class. We can also use, uh, some more crazy— I think this is something genetic something, uh, I— but basically weird statistical patterns here that give you electric guitar or baseball with very high confidences. Okay. Um, so that's kind of weird. I'd like to point out before we dive into exactly what's going on here that, uh, these kinds of results are actually not super new to computer vision. There are some papers, for example, from 2011 where you had similar kind of depressing results. Well, for example, they had two images here and they found that they could change one image into an image that looks to us very similar. But in fact, these HOG representations that we really liked to extract before ConvNets came about, They are in fact identical for these two images, even though this image is clearly— they look very similar, but their HOG representation, the feature representation, is identical. So there were a few papers that were trying to study exactly what's going on with these, as we call, adversarial examples. One of the better papers on this topic was Explaining and Harnessing Adversarial Examples from last year by Ian Goodfellow, Jon Schlanz, and Christian Sagety. They actually concluded from a whole bunch of experiments, and I encourage you to read through this paper, that actually the primary cause of this vulnerability is the linear nature of the functions that we're using in the forward pass. And we'll unpack this over time, I think by the end of this lecture hopefully. Um, but just the intuition I'd like to give you is the following. Images are super high-dimensional objects, right? Lots of pixels. So like 150,000-dimensional space or something like that. And the real images that we actually train on, they have special statistical structure and they're constrained to a tiny manifold of this space. And we're training ConvNets on this, and those ConvNets work extremely well on that tiny manifold where the statistics of the images are image-like. But, um, we're putting these linear functions on top of it and, uh, we're correctly classifying that tiny sub-manifold, but outside of that manifold, we're kind of casting these shadows of complete randomness and chaos because we we haven't trained there. What I mean by that is suppose here we have a simple example and I try to classify this with a two-layer net, and you can see that, uh, around the data points we're classifying them correctly, but look at what I call shadows. These are linear functions that are generalizing in very funny way outside of the data manifold. So we are casting these linear weird shadows outside of the data manifold, like all bets are out. We haven't trained there, we haven't imposed anything there, and so arbitrary things happen outside of your data manifold. So anytime we're— so, um, so that's something to keep in mind. And imagine this not in the space of 2 dimensions. Imagine it in a space of 150,000 dimensions and how these shadows, as I like to call them, like, interact in very funny ways. So let's see a concrete example of just exactly how this linear nature might be a problem and what might be going on. We'll work with a specific case of logistic regression. Um, so we're now in a case of just very simple Logistic regression in small space. So we'll be doing W transpose X plus B, where X is, say, like, uh, 10-dimensional in this example we're about to go into. W is a small 10-dimensional vector and B is a bias. And in logistic regression, we are putting that through a sigmoid. So this is the raw score we're putting through a sigmoid, and we're interpreting the output of that sigmoid as the probability that this input X is of class 1. Okay? That's what this says. So basically, we're computing a score here from the linear classifier, And then we're interpreting this to be class 1 if the score is greater than 0, or equivalently, if what comes out of the sigmoid is greater than 0.5, because that's indicating that the probability of 1 is greater than 50%. So that's the setup for logistic regression. And now we're going to go into a specific example here where we have some example x, and we have some weights w. And we plug this into our classifier, and we're going to compute w transpose x plus b. I guess we don't have a bias in this particular this particular case. But we're going to compute W transpose X, which is really just 2 times negative 1 plus negative 1 times negative 1 times 3 times 1 and so on, right? So all of that multiply through just a dot product of those two vectors. We see that the score that comes out is negative 3. And then when we put that through the sigmoid, that's telling us that with the current setting of weights, this classifier thinks that you have probability of class 1 of 0.0474. So in other words, with this setting of weights, this classifier is 95% certain that this is a class 0 example. And now what we're going to do is we're going to try to construct an adversarial example in the following sense, that we'd like to construct an X that is very similar to this X, but we'd like it to, uh, be classified as 1. So right now, the classifier thinks there's 95% chance of being class 0. We, we're going to try to flip that probability the other way around by slightly modifying X. And so if you have access to the weights, which in this case we do and we know exactly what they are, you can in fact come up with a technique for actually constructing this adversarial X. And, uh, so how would we nudge X in every single dimension to screw up that classifier to make it predict, uh, class, uh, the other class? For example, how would we change in the first column? What would— what do we wanna do with X in the first column in order to screw up the classifier? Way too many voices. Uh, yes. So we want to make only tiny changes. The idea here is the adversarial example wanted to be very close to x. Um, wanted to be very close to x. So we want to slightly depart from 2. You want to decrease x. You want to make x smaller because then this thing will become smaller contribution. So you're gonna get a smaller number than -2, in this dot product. So in fact, we can do this in every single column independently. So we want to decrease in here, we want to decrease in the second column, we want to slightly increase in the third column, and so on. So I can construct an adversarial example because I know what the weights are. And in particular, I'm going to, uh, only change it by 0.5 in every single dimension, but exactly in the correct way. And when I do that, what I find is when I do the dot product with my adversarial x, then since I've made tiny change in every single dimension, all of these changes exactly added up in the constructive way to give me suddenly a score of 2. And when you put that in the softmax— in the, sorry, the sigmoid, you come up to 88%. So I've slightly changed my x and I've blown up the probability. And truly the reason this happened is because I was able to make a tiny change in every single dimension because I know what the weights are, and that allows me to blow up, uh, this score. And so think about— this is just a small example with 10 dimensions, but think about images which are 224 by 224, say. So there's 150,000 numbers, and the gradient tells me exactly how to change every single pixel in order to do arbitrary things downstream, like increase the probability of ostrich. And since I know what that gradient is on my image, I can nudge every single one of these 150,000 numbers in exactly the correct way by a tiny amount, and that allows me to completely blow I can blow up any class I want in a slightly depressing manner. And so that's roughly what's going on and why this is kind of happening. And so I'll reinterpret this again in a bit. I'd just like to point out that— so basically, the linear classifiers are a problem. They're very nice to use because they're easily trainable, and ConvNets work very well with them in the data manifold. But they kind of— they're not very nice functions because you end up doing these very large dot products. and it's a lot of space to do small changes with large effects if you know exactly what changes to make. And, uh, I'll come back to this point in a bit. I'd just like to show you that actually we can do very similar things and produce adversarial examples just with linear classifiers. So this has nothing to do with deep learning or with convolutional networks. We can do this with linear classifiers. So, um, so here we have these templates for CIFAR-10. We can also train linear classifiers in the case of ImageNet, So these are linear classifiers, templates that we're familiar with for, say, pizza or for daisies or for, uh, plates and so on. Okay? So we're just doing linear classification ImageNet, and I'm going to construct adversarial examples. And so for example, if I take a random noise pattern and my classifiers think that this is, uh, 1% of being a bobsled, but basically very unconfident prediction here, I can take— since I know the weights for a goldfish classifier, I can add a tiny bit— I can mix in a tiny bit of goldfish weights, and I get an image that looks identical, although it's hard to see at this point. But this is 100% goldfish because I changed every single pixel a tiny amount in exactly the right direction. I can do this also with real images. So I can take this image. I can add in a tiny bit of goldfish. And it turns out that I can make this much more likely goldfish. I can make the goldfish class most likely just by mixing in a tiny bit of those weights. Okay? And I can also do that, for example, with, uh, with an actual goldfish and I can make it a daisy, although this is not as clear. And so really what's going on here is— what's interesting is that, um, some of these adversarial examples kind of came about first in the setting of ConvNets applied to images, and that's where we first noticed that this is happening. But in fact, this has nothing to do with ConvNets because you can do it with linear classifiers. And, and in fact, like deep learning, the fact that we have all this wiggle, that that will allow us to maybe fix the problem, although we— I don't know exactly how we'll have to change the linear classifier, how we use linear functions might have to change. And also I wanted to make the point that this has nothing to do with images. We can do this in any other modality. So speech recognition is subject to the exact same problem because we're using these large networks made up of linear functions in between, and we can blow up these scores anyway. And so basically the way to think about this is on the data manifold where we train our images with their specific statistical structure, everything works. But since these things are so high-dimensional and we have the gradient, uh, and this gradient, by the way, is like the optimal way of— the gradient tells you the optimal way of screwing up whatever you like, and you're computing it with this dynamic programming, which is really backpropagation, is a dynamic programming way of figuring out how to change your image in the optimal way to do anything you like downstream. And so in these high-dimensional spaces, we can just get the precisely correct direction and we just make small steps in it, and we can kind of get to arbitrary results. Once we actually meddle with the low-level statistical structure of those images. And so that's kind of what's going on, and we need to find a way to fix it. And right now there aren't very good answers to it, I think, but it's a subject of research, area of subject. Good? [Speaker:AUDIENCE] I mean, could you, after you have a fairly well-trained network, do some data augmentation, you know, take these perturbed examples, feed them back in? [Speaker:ANDREJ KARPATHY] Yeah. So people have investigated a whole bunch of kind of simple fixes to this. So one example is if we don't just classify a single image, but we actually do many crops and we classify all those, you become more resistant to some of this for sure. But for example, some people have tried simple, simple ways of fixing it. Like for example, I train my ConvNet, I generate an adversarial example, and then I can add it into my training data as this is like a negative class or something like that. And I keep repeating this and it turns out that you can always find more adversarial examples. Like it doesn't actually end up working very well. And so there's a few things that people have played with which unfortunately I don't have time to go into by the end of this lecture. But basically, small hacks that people have tried, and nothing so far is very convincingly working. Something that is working is taking out the linear functions and trying different things. And so you become much more robust to the adversarial stuff, but then your classifiers don't work as well as they used to. And so there's some stuff like that, some trade-offs. Any— that? [Speaker:AUDIENCE] Why is this a big thing? Yeah. Uh, so why is this a big problem? Uh, maybe one, um, so first of all, it's very depressing and kind of confusing. So that's a problem by itself. But in practice, maybe why it's a problem, for example, we found that these adversarial examples, in fact, they're not just a function of a particular ConvNet, but they kind of transfer from one ConvNet to another, another one that was trained completely separately on different, like, even data or so. So something on the low-level structure of that we're messing with images here is kind of screwing up ConvNets all over the place. Um, and so adversarial examples in one domain are likely to be adversarial in, in the different ConvNet. And so maybe, for example, all of the companies are working with ConvNets right now and analyzing your pictures with them. Like, uh, you don't actually want people to be able to mess with your processing, maybe. I don't know. That's one example. Or I can maybe take my image and I can convince So I know that this will go into Google and they'll analyze it with their ConvNets. And I can make my image adversarial on my ConvNet to be whatever I want. And then maybe I upload it and has maybe— there's some chance that the same thing will carry over. And so maybe there's some repercussions to this, but I haven't fully thought it through. But I think it's definitely worrying. Go ahead. Could you maybe show an image to a self-driving car that has— [INAUDIBLE] Yeah, so that's a good example. I don't think that would work, right? Because this is like you're really meddling with the precise pixels that go into the ConvNet. Like if you just show it with the camera, you again, you don't have these weird physical patterns. So I doubt that that would work. If someone wants to do that for a project, by the way, adversarial examples carry over to like pictures of it, that would be really cool. I doubt that that would work. Go ahead. What about face recognition? You can use like logging in computers and things like that. Oh, I see. That's a very interesting example. Face recognition. So I can make an adversarial example to be anyone I want, and then— [LAUGHTER] Yeah, but again, you would have to show that— yeah. So I should say that these examples don't just transfer right away. There's like a small chance, higher than you'd think by chance, that they do transfer, which is slightly worrying. Go ahead. [Speaker:AUDIENCE] Is it prefixable in the sense that it's kind of like your case of curse of dimensionality? Because they're like these images are kind of out of sample, no? Yeah, so the images are very out of sample, as you mentioned. And you still wouldn't like this property to hold true. Like, why can't you still recognize the— if you can recognize the school bus, you'd like to think that the ConvNet should as well. I think that the— but we do have to change our training process or something about it. And it's not clear right now what to do. Yeah, maybe add a lot of noise. And say this is class of non-picture or something. Yeah. Anyway, so since it's 4:19, I'd just like to summarize that backdropping to the image is a very powerful technique. And we saw many ways that this can be used for understanding, for segmenting, kind of, for inverting, for fun, and for confusion and chaos definitely as well. And so next lecture, we're going to go into recurrent neural networks, current neural network language models and image captioning and stuff like that. So that should be fun. It is 4:20, so I'll end the class lecture here. But you can come.