Jason Yosinski shows how to fool deep neural networks by generating fake images that networks confidently misclassify. Using gradient ascent to optimize pixels, researchers created images of gorillas and buses that don't look like those things to humans but fool the network completely.
Chapters
So we had hoped to generate beautiful pictures of these different classes. In fact, all we found is something completely different, unexpected. Basically, our conclusion was deep neural networks are easily fooled. It's easy to find these types of crazy pictures, noisy pictures— they're now called fooling examples or adversarial examples— that cause the network to have, like, extreme responses in different ways. So we're going to ask another question. So I've shown you something about what happens in the middle of this network, the firing's going on. Let's ask another related question about this, maybe this gorilla picture that we're putting into the network, which is, what makes a gorilla a gorilla to the network? So I show you this picture of a gorilla kind of sideways. Sorry. I show you this picture of the gorilla. You kind of know it's a gorilla. But what about it, like what specifically about that picture makes it a gorilla to the network? Is it, is it like the face of the gorilla, or the fur of the gorilla, Or is it the background? Like, any animal on top of this specific type of grass near gorilla habitats, it'll say is a gorilla. So we'd like to kind of get the network to draw us a picture of what it thinks a gorilla is. And this started like this whole other path of papers, which I thought was pretty fun. I'll show you a little bit about it now. To do that— so we want the network to paint this gorilla for us. So what we do is we actually give it— we start it with not a real picture of a gorilla, But we started with some noise. So we put some, like, pixely noise in. We do forward prop. And rather than saying gorilla, because that's not a gorilla, the network tends to say, I'm not sure what that is. Lots of little neurons fire somewhat. And then we say, ah, OK, you say the amount of gorilla there is like 1%. I'd like you to find more gorilla in this picture. So plus gorilla, please. Oh, OK. We use something called backprop to compute in pixel space the little tiny pixel changes we would have to make to make the network fire more next time. So with these pixels, it fires, like, say, 0.01 gorilla. If we change the pixels, we adjust this one a little brighter, a little less bright, a little more green, a little more red, maybe the next time we can make it fire a little bit more. So we do that. We take this picture, we adjust it a tiny bit, and we try to get it to say gorilla a little bit more. We do that in like a for loop in Python, and we just like iterate it 200 times to make it like— to adjust the picture a little bit, a little bit, a little bit, a little bit, to hopefully make it fire more and more. And at the end, we output the picture and see what we see. And so here's what we saw. Here's the first, you know, run of the for loop 200 times. This is optimized for gorilla. So to us, we saw this and we thought, hmm, we were hoping for a beautiful gorilla or like an abstract art gorilla showing just the outline if that's what's important. But we don't see that at all. We see all these, like, crazy pixels. So first we thought maybe our code is broken. Maybe we start at 0.01 gorilla and we're basically getting to, like, 0.02 gorilla. We're not really getting anywhere. But actually, we feed this network— this image to the network, and it says this is 0.9999 gorilla. So 99.99%, the network is sure this is gorilla. Thought that was pretty interesting. We tried the for loop again and ran many more times. We can make many more kind of fake non-gorilla gorillas. We tried it for different classes. So here's— the network thinks this is a cliff dwelling or a parking meter or a Windsor tie, all above 99.99%. So we had hoped to generate beautiful pictures of these different classes. In fact, all we found is something completely different, unexpected. Basically, our conclusion was deep neural networks are easily fooled. It's easy to find these types of crazy pictures, noisy pictures. They're now called fooling examples or adversarial examples that cause the network to have, like, extreme responses in different ways. Okay. So this is— this was, like, the first paper in this direction. We tried— this is without regularization. You get these crazy patterns. We tried adding more regularization, so, like, making pushing these pixels toward more realistic patches of images. Some other folks did this at the same time as well. So this is using L2 regularization. This is keeping the images near the origin. This is a learned regularizer, which is a little bit more complicated. In both of these cases, you can kind of see a little bit of birds emerging or peacocks emerging. After playing around with it for quite a while, we found a different combination of regularizers, so ways of running this for loop in Python, That resulted in much more clear visualizations. So here we can now draw, like, this is what the network thinks of as a hartebeest or a school bus here. So now rather than those pixels, we can kind of see, ah, the network thinks when we say hartebeest, what the network thinks is a couple features actually. So first you can see these like curvy horns. So somehow hartebeests actually are animals with curvy horns. The network thinks the horns are important. Makes sense, because they're— the curvy horns are kind of a unique feature of hartebeest. It also knows that there's sort of, like, green in the background and that they tend to be kind of brown. On the other hand, you don't really see hartebeest hooves, right? So the network doesn't really think of, like, the important part of hartebeest is having hooves, probably because it's trained to recognize so many other animals with hooves. So hooves are not really special to hartebeests. It's like we talked about before with the dogs, the different species of dogs. If you train the network to recognize 100 different species of dogs— it'll have to learn like these little features of the, the hair and the ears and the eye shapes and so on. If you just lump all the dogs together though, any simple feature that makes it a dog instead of a cat will suffice. Cool. So we can try this for other things too. If we visualize like the output layers, we visualize the last layer neurons, we can basically visualize the classes that the network knows about. And fortunately, we know about those too. So here's what it thinks a pirate ship is, or a teddy bear, or a pitcher. We can also ask it to visualize, like, intermediate neurons. So if we visualize the 2 layers before that, 6 and 7, we see kind of a crazy amalgamation of different patterns. We actually don't really understand what we see here very well yet. However, if we visualize lower layers, so all the way down to the bottom, here's these patterns I showed you before, these very simple, like, edges. Like things changing from light to dark, green to red, something like that. One layer up, you can see it's combining these simple patterns into slightly more complicated patterns. So here's— you see corners and edges and curved things. Layer 3, 4, 5. By layer 5, we see some pretty cool stuff. We see, for example, the face detector is not on here, but there was that face detector. We see also here's a detector for like legs emanating from a central area, so maybe for octopus or spider or something. We see here like a 3-dimensional corner. There's some water. We can detect water. This is actually the text detector. It's pretty hard to see. If you zoom in— this is like a big picture. If you zoom into the center, you'll see actually what it's looking for when it detects text isn't like A, B, C, D, E, F. It's simply a relative, like, vertical and horizontal spatial frequencies. That happen to correspond roughly to text. So it's good to know our network can't actually read, can't read English, or for that matter other languages. But by knowing that it's looking for these simple patterns, we can actually kind of assume if we showed it Latin instead of English, it would respond just about as much. So by taking all these approaches, all these things, like looking at it from all these different angles, we can start to understand more and more about what neural networks are actually doing.
