CS231n Winter 2016: Lecture 8: Localization and Detection

3 o'clock, so let's get started. So I'm going to lecture today, give Andrej Karpathy a little bit of a break. So today we're— so last time we talked about sort of we saw all the parts of ConvNets. We put everything together. Today we're going to see some applications of ConvNets to actually dive inside images and talk about spatial localization and detection. We were— we actually moved this lecture up a little bit. We had it later on the schedule. But we saw a lot of you guys were interested in this stuff for projects, so we wanted to move it earlier to kind of give you an idea of what's feasible. So first, couple administrative things. The project proposals were due on Saturday. My inbox kind of exploded over the weekend, so I think most of you submitted it. But if you didn't, you should probably get on that. We're in the process of looking through those, so we'll go to make sure that the project proposals are reasonable and that everyone submitted one. So we'll hopefully get back to you on your projects this week. Also, homework 2 is due on Friday. So who's, who's done? Who's stuck on batch norm? Okay. Good. Good. That's fewer hands than we saw last week. So we're making progress. Also keep in mind that we're asking you to actually train a pretty big ConvNet on CIFAR for this homework. So if you're starting to train on Thursday night, that might be tough. So maybe start early on that last part. Also Homework 1, we're in the process of grading. Hopefully we'll have those back to you this week, um, so you can get feedback before Homework 2 is due. Also keep in mind the— we actually have an in-class midterm next week on Wednesday. Um, so that's a week from Wednesday. So be, be ready. That'll be in class. It should be a lot of fun. All right. So last lecture we were talking about convolutional networks. We kind of saw the pieces. We spent a long time understanding how this convolution operator works, how we're sort of transforming feature maps from one to another by running inner products over— by sliding this window over our feature map, computing inner products, and actually transforming our representation through many layers of processing. And if you remember, these lower layers of convolutions tend to learn things like edges and colors, and higher layers tend to learn more complex object parts. We talked about pooling, which is used to subsample and downsize our feature representations inside networks. So that's a common ingredient that we saw. We also did case studies on particular ConvNet architectures so you could see how these things tend to get hooked up in practice. So we talked about LeNet, which is something from '98. It's a little 5-layer ConvNet that was used for digit recognition. We talked about AlexNet that kind of kicked off the big deep learning boom in 2012 by winning ImageNet. There was an 8-layer ConvNet. We talked about ZFNet that won ImageNet classification in 2013. It was pretty similar to AlexNet. And then we saw that deeper is often better for classification. We looked at GoogleNet and VGG that did really well in 2014 competitions that were much, much deeper than AlexNet and ZFNet and did a lot better. And we also saw this new fancy crazy thing from Microsoft called ResNet that that won just in December in 2015 with 150-layer architecture. And as you'll recall, just the— over the last couple of years, these different architectures have been getting deeper and getting a lot better. But this is just for classification. So now in this lecture, we're going to talk about localization and detection, which is actually another really big important problem in computer vision. And this idea of deeper networks doing better, hint, that'll kind of— we'll revisit that a lot in these new tasks as well. So, so far in the class, we've really been talking about classification, which is sort of given an image, we want to classify which of some number of object categories it is. That's a nice basic problem in computer vision that we've using— that we're using to understand ConvNets and such. But there's actually a lot of other tasks that people work on too. So some of these are classification and localization. So now instead of just classifying an image, as one of— as some category labels. We also want to draw a bounding box in the image to say where that class occurs. Another problem people work on is detection. So here there's again some fixed number of object categories, but we actually want to find all instances of those categories inside the image and draw a box around them. Another more recent task that people have started to work on a bit is this crazy thing called instance segmentation, where again you want— you have some fixed number of object categories, you want to find all instances of those categories in your image, But instead of using a box, you actually want to draw a little contour around and kind of identify all the pixels belonging to each instance. Instance segmentation is kind of crazy, so we're not going to talk about that today. Just thought you should be aware of it. And we're going to really focus on these localization and detection tasks today. And the big difference between these is the number of objects that we're finding. So in localization, there's sort of one object or, in general, a fixed number of objects. Whereas in detection we might have multiple objects or a variable number of objects. And this seems like a small difference, but it'll turn out to actually make a big— be pretty important for our architectures. So we're going to first talk about classification and localization because it's kind of the simplest. So just to recap what I just said, classification, one image to a category label. Localization is image to a box. And classification-localization means we're going to do both at the same time. Uh, just to give you an idea of the kinds of datasets people use for this, um, we to— we've talked about the ImageNet classification challenge. Uh, ImageNet also has run a classification plus localization challenge. So here, um, similar to the classification task, there's 1,000 classes. Um, and each training instance in those classes actually has one class and several bounding boxes for that class inside the image. And now at test time, your algorithm again makes 5 guesses. Where instead of your guesses just being class labels, it's a class label together with a bounding box. And to get it right, you need to get the class label right and the bounding box right, where getting a bounding box right just means you're close in some thing called intersection over union that you don't need to care about too much right now. So again, you get it— for ImageNet at least, you get it right if one of your 5 guesses is correct. And this is kind of the main dataset that people work on for classification plus localization. So one really fundamental paradigm that's really useful when thinking about localization is this idea of regression. So I don't know, thinking back to a machine learning class, you kind of saw, like, classification, like maybe with an SVM, and regression, maybe with linear regression or something fancier. And when we're talking about localization, it's really like— we can really just frame this as a regression problem where we have an image that's coming in. That image is going to go through some processing. And we're eventually going to produce 4 real-valued numbers that parameterize this box. There's different parameterizations people use. Common is xy-coordinates of the upper left-hand corner and the width and height of the box. But you'll see some other variants as well. But always 4 numbers for a bounding box. And then there's some ground truth bounding box, which again is just 4 numbers. And now we have— we can compute a loss, like maybe Euclidean loss is a pretty standard choice, between the numbers that we produced and the correct numbers. And now we can just train this thing just like we did our classification networks, where we sample some mini-batch of data with some ground truth boxes, we propagate forward, compute a loss between our predictions and the correct predictions, backpropagate, um, and just update the network. So this paradigm is, is really easy. This actually makes this localization task, uh, actually pretty easy to implement. So here's a really simple recipe for how you could implement classification plus localization. So first, you just download some existing pre-trained model, or you train it yourself if you're ambitious. Something like AlexNet, VGG, GoogleNet, all these things we talked about last lecture. Now, we're going to take those fully connected layers that were producing our class scores. We're going to set those aside for the moment, and we're going to attach a couple new fully connected layers to some point in the network. This will be called— we'll call this a regression head. But I mean, it's basically the same thing. It's a couple fully connected layers, and then it outputs some real-valued numbers. Um, now we train this thing just like we trained our classification network. The only difference is that now instead of class scores, um, and ground truth classes, we use, uh, L2 loss and ground truth boxes. Other than that, we train this network exactly the same way. Now at test time, we just use both heads to do classification and localization. We have an image, we've trained the classification heads, we've trained the localization heads, We pass it through, we get class scores, we get boxes, and we're done. Like, really, that's, that's all you need to do. So this is kind of a really nice simple recipe that you guys could use for classification plus localization on your projects. So one slight detail with this approach, there's sort of two main ways that people do this regression task. You could imagine a class agnostic regressor or class specific regressor. you could imagine that no matter what class, I'm gonna use the same architecture, the same weights in those fully connected layers to produce my bounding box. So that would mean you're sort of outputting always 4 numbers, which are just the box no matter the class. Um, an alternative you'll see sometimes is class-specific regression, where now you're gonna output C times 4 numbers. That's sort of like 1 bounding box per class. Um, and different people have found that sometimes these work better in different cases. But I mean, intuitively, it kind of makes sense that some— that the way you might think about localizing a cat could be a little bit different than the way you localize a train. So maybe you want to have different parts of your network that are responsible for those things. But so then it's pretty easy. Then you just— it changes your bet— the way you compute loss a little bit. You compute loss only using the ground truth class— the box for the ground truth class. But other than that, still basically the same idea. Another design choice here is where exactly you attach the regression head. Um, again, this isn't too important. Different people— you'll see different people do it in different ways. Some common choices would be to attach it right after the, the last convolutional layer. Um, that would just sort of mean like you're re-initializing new fully connected layers. Um, you'll see things like Overfit and VGG localization work this way. Another common choice is to just attach your regression head actually after the last fully connected layers from the classification network. And you'll see some other things like DeepPose and RCNN kind of work in this flavor. But either one works fine. They'll— probably you could attach it just about anywhere, and it would do something. So as an aside, we can actually generalize this framework to localizing more than one object. So normally with this classification localization task that we sort of set up on ImageNet, we care about producing exactly one object bounding box for the input image. But in some cases, you might know ahead of time that you always want to localize some fixed number of objects. Um, so here, this is really easy to generalize. Now, your regression head, it just outputs a box for each of those objects that you care about. And again, you train the network in the same way. Um, and this idea of actually localizing multiple objects at the same time is pretty general and pretty powerful. So for example, this kind of approach has been used for human pose estimation. So the idea is we want to input a close-up view of a person, and then we want to figure out what's the pose of that person. So well, people sort of generally have a fixed number of joints, like their wrists and their neck and their elbows and that sort of stuff. So we just know that we need to find all the joints. So we input our image. We run it through a convolutional network. And we regress x, y coordinates for each joint location. And that gives us our— that actually lets you predict a whole human pose using this sort of localization framework. I mean, this paper— and there's a paper from Google from a year or two ago that does this sort of approach. They had a couple other bells and whistles, but the basic idea was just regressing using a CNN to these joint positions. So overall, this idea of localization and treating it as regression for a fixed number of objects is really, really simple. So I know some of you guys on your projects have been thinking that you want to actually run detection because you want to understand, like, ins— parts of your images or find parts inside the image. And if you're thinking of a project along those lines, I'd really encourage you to think about this localization framework instead. That if there's actually a fixed number of objects that you know you want to localize in every image, you should try to frame it as a localization problem because that tends to be a lot easier to set up. All right. So actually, this simple idea of localization via regression actually is really simple. It'll actually work. I would really encourage you to try it for your projects. But if you want to win competitions like ImageNet, you need to add a little bit of other fancy stuff. So another thing that people do for localization is this idea of sliding window. So we'll step through this in more detail. But the idea is that you, you still have your classification localization dual-headed network. But you're actually going to run it not once on the image, but at multiple positions on the image. And you're going to aggregate across those different positions. And you can actually do this in an efficient way. So to sort of see more concretely how this sliding window localization works, we're going to look at the OverFeat architecture. So OverFeat was actually the winner of the ImageNet localization challenge in 2013. It— this sort of setup looks basically like what we saw a couple slides ago. We have, um, an AlexNet at the beginning, then we have a classification head, we have a regression head. Classification head is spitting out class scores, regression head is spitting out bounding boxes. Um, and this thing, because it's an AlexNet type architecture, is expecting an input of 221 by 221. Um, but actually, we can run this on larger images, and this can help sometimes. So suppose we have a lar— a larger image, —what did I say—257 by 257. Now, we could imagine taking our classification plus localization network and running it just on the upper corner of this image. And that'll give us some class score and also some regressed bounding box. And we're going to repeat this, take our same classification plus localization network and run it on all 4 corners of this image. And after doing so, we're going will end up with 4 regressed bounding boxes, one from each of those 4 locations, together with a classification score from each location. But we actually want just a single bounding box. So then they use some heuristics to merge these bounding boxes and scores. And that part's a little bit ugly. I don't want to go into the details here. They have it in the paper. But the idea is that probably combining, aggregating these boxes across multiple locations can help the model sort of correct its own errors. And this tends to work really well. And I mean, it won them the challenge that year. But in practice, they actually use many more than 4 locations. Oh, yeah, question? [Speaker:AUDIENCE] So on the previous slide, so each of those red boxes, they weren't always within the slice being looked at. Is that just a design of the slide? Like, normally the box would be fully within the— [Speaker:ANDREJ KARPATHY] Well, I mean, yeah, yeah, that's actually a good point. So once you're doing regression, you're just predicting 4 numbers. You could in theory predict anywhere. It doesn't have to be inside the image. Although that brings up a good point. When you're doing this, especially when you're training this network in this sliding window way, you actually need to shift the ground truth box in a little— you need to shift the coordinate frame for those different slices. That's kind of an ugly detail that you have to worry about. Yeah, but in practice, they use many more than 4 image locations. And they actually do multiple scales as well. So you can see this is actually a figure from their paper. On the left, you see all the different positions where they kind of evaluated this network. In the middle, you see those output regressed boxes, one for each of those positions. On the bottom, you see the score map for each of those positions. And then, I mean, they're pretty noisy, but it's kind of conver— they're kind of generally over the bear. So they'd run this fancy aggregation method, and they get a final box for the bear, and they decide that this image is a bear. And they actually won the challenge with this. But one problem you might anticipate is it could be pretty expensive to actually run the network on every one of those crops. But there's actually a more efficient way— thing we could do. So we normally think of these networks as having convolutional layers and then fully connected layers. But when you think about it, a fully connected layer is just 4096 numbers, right? It's just a vector. Um, but instead of thinking of it as a vector, we could think of it as just another convolutional feature map. It's kind of crazy. We've just transposed it and added two 1x1 dimensions. So now the idea is that, um, our— we can now treat our, our fully connected layers and convert them into convolutional layers. Because if you imagine, um, in our fully connected network, we had this, uh, convolutional feature map, and we had one weight from each element of that convolutional feature map to produce each element of our 4096-dimensional vector. Um, but we— instead of thinking about reshaping and having an affine layer, that's sort of equivalent to just having a 5x5 convolution. It's a little bit weird, but if you think about it, it should make sense eventually. But— right, so then we take this fully connected layer, turns into a 5x5 convolution. Then we previously had another fully connected layer going from 4096 to 4096. This is actually a 1x1 convolution. Right? That's kind of weird, but if you think hard and sort of work out the math on paper and go sit in a quiet room, you'll figure it out. And so we basically can turn each of these fully connected layers in our network into a convolutional layer. And now, now this is pretty cool because now our network is composed entirely of just convolutions and pooling and element-wise operations. So now we can actually run the network on images of different sizes. And this sort of will give us very cheaply equip— the equivalent of operating the network independently on different locations. So to kind of see how that works, um, you imagine at training time, you're maybe working over a 14 by 14 input. You run some convolutions, and then here are our fully connected layers that we're now reimagining as convolutional layers. So then we had this 5 by 5 conv block that gets turned into these 1 by 1, uh, spatially sized elements. So we've— we're sort of eliminating— not showing the depth dimension here, but these— like this 1 by 1 would be 1 by 1 by 4096. Right? So we're just converting these layers into convolutional layers. And now that we— now that they're convolutions, we could actually run on an input of a larger size. And you can see that now we've added a couple extra pixels to the input. And now we actually run all these things as convolutions and get a 2x2 output. But what's really cool here is that we're able to share computation to make this really efficient. So now our output is 4 times as big, but we've done much less than 4 times the compute. 'Cause if you think about the difference between where we're doing computation here, the only extra computation happened in these yellow parts. So now we're actually very efficiently evaluating the network at many, many different positions without actually spending much computation. So this is how they are able to evaluate that network in that very, very dense multiscale way that you saw a couple slides ago. Does that make sense? Any questions on this? Okay, good. Right? So actually, we can look at the classification plus localization results on ImageNet over the last couple of years. So in 2012, Alex Kruchevsky, Jeff Hinton, Ilya Sutskever, they won not only classification but also localization. But I wasn't able to find any published details of exactly how they did that. In 2013, it was the OverFeat method that we just saw actually improved on Alex's results a little bit. The year after, we talked about VGG and their sort of really deep 19-layer network. They got second place on classification, but they actually won on localization. And the VGG actually used basically exactly the same strategy that OverFeat did. They just used a deeper network. And actually, interestingly, VGG used fewer scales. They stamped that network out in fewer places and used fewer scales. But they actually decreased the error quite a bit. So basically, the only difference between OverFeat and VGG here is that VGG used a deeper network. So here we could see that these really powerful image features actually improve the localization performance quite a bit. We didn't have to change the localization architecture at all. We just swapped out a better CNN, and it improved results a lot. And then this year in 2015, I mean, Microsoft swept everything, as that'll be a theme in this lecture as well. This, this 150-layer ResNet from Microsoft crushed localization here and dropped performance from 25 all the way down to 1%. All the way down to 9. But I mean, this is a little bit fan— this is tough to really isolate the deep features. So yes, they did have deeper features. But Microsoft actually used a different localization method called RPNs, region proposal networks. So it's not really clear whether this— which part— whether it's the better localization strategy or whether the better features. But at any rate, they did really well. So that's pretty much all I want to say about classification localization. Just consider doing it for your projects. And if there's any questions about this task, We should talk about that now before moving on. Yeah? [Speaker:AUDIENCE] How much are you limited by outliers in your dataset? So if ImageNet has things that are incorrectly labeled or the boxes are in the wrong places? [Speaker:ANDREJ KARPATHY] Yeah, that can always hurt performance, especially with an L2 loss. So then L2 loss and having outliers is actually really bad. So sometimes people don't use an L2 loss. Instead, you can try an L1 loss. That can help with outliers a little bit. People also will do sometimes a smooth L1 loss where it looks like L1 sort of at the tails, but then near 0, it'll be quadratic. So actually swapping out that regression loss function can help a bit with outliers sometimes. But also if you have a little bit of noise, sometimes hopefully your network will just figure it out. Cross your fingers. Don't think too hard. Any other questions? Yeah. When we're using one of these pretrained nets and just training the regression head, —call it. Do we just backprop into our own graph net, or can we backprop all the way into the, uh, the language? Yeah. Yeah, good question. So people do both, actually. Um, so for Overfeat, uh, actually, I don't remember. I don't remember exactly which Overfeat did, but VGG actually backprops into the entire network. Um, so it'll be, it'll be faster to just— it'll actually work fine if you just train the regression head, but you'll tend to get a little bit better results if you backprop into the whole network. And VGG did this experiment, and they got maybe 1 or 2 points extra by backpropping through the whole thing. but at the expense of a lot more computation and training time. So I would say, like, as a first thing, don't— just try not backproping into the network. Does that even generalize if— I mean, the testing data can sometimes look pretty different from what these nets were trained on, right? Well, generally not, right? Because you're testing them on the same classes that you saw at training time. So you're going to see different instances, obviously, but I mean, you're still— you saw bears at test time, you're going to see bears at training time. We're not expecting you to generalize across classes. That would be pretty hard. Yeah. Um, if you, if you're backproping a whole network, does that mean you then have to train with the classification hit and the localization hit simultaneously? Ah, good question. Yeah. So sometimes people will do that. They'll train with both simultaneously. Also, sometimes people will just end up with separate networks, one that's sort of only responsible for regression, one that's only responsible for classification. Um, those both work. What if you don't know the number of objects that are going to appear in the image? Well, glad you asked. That's actually the next thing we're going to talk about. That's a different task called object detection. So yeah. Is there synchronization between the localization class and then the classification class specifically? Well, uh, yeah. Well, so I mean, it kind of depends on the training strategy. If you're like— if you— it also kind of goes back to this idea of class agnostic versus class-specific regression. So if you're doing class agnostic regression, it doesn't matter. You just regress to the boxes no matter the class. Class specific, you're sort of training separate regressors for each class. All right. Let's talk about object detection. So object detection is, is like much fancier, much cooler, but also a lot hairier. So the idea is that, again, we have an input image. We have some set of classes. We want to find all instances of those classes in that input image. So I mean, you know, regression worked pretty well for localization. Why don't we try it for detection too? Well, because in this image we have these dogs and cats, and we have 4 things. We have 16 numbers. That looks like regression, right? Image in, numbers out. But if we look at another image, then, you know, this one only has 2 things coming out. So it has 8 numbers. If we look at this one, there's a whole bunch of cats. So we need a bunch of numbers. So I mean, it's kind of hard to treat detection as straight up regression because we have this problem of variable sized outputs. So we're going to have to do something fancier. Although actually there is a method we'll talk about later that sort of does this anyway and does treat it as regression, but we'll get to that— we'll get to that later. But in general, you want to not treat this as regression because you have variable sized outputs. So a really easy problem— a really easy way to solve this is to think of detection not as regression, but as classification. In machine learning, regression and classification are your two hammers, and you just want to use those to beat all your problems. So regression didn't work, so we'll do classification instead. We know how to classify image regions. We just run it through a CNN. So what we're going to do is we're going to take many of these input regions of the image, put a classifier there, and say, oh, right, this region of the image, is it a cat? No. Is it a dog? No. Move it over a little bit. Oh, we found a cat. That's great. Move it over a little bit. That's not anything. So then we can actually just try out a whole bunch of different image regions, run a classifier in each one, and this will basically solve our variable-sized output problem. So there's— oh, yeah, question? [Speaker:AUDIENCE] So how do you decide what's the window size? [Speaker:ANDREJ KARPATHY] So the question of how to decide what the window size, the answer is we just try them all, right? Just literally try them all. So that's actually a big problem, right? Because we need to try windows of different sizes, of different positions, of different scales. We need to do this for every test image. This is going to be really expensive, right? There's a whole lot of places we need to look. Yeah? [Speaker:AUDIENCE] Also here you're saying picking your windows, does that look like anything? But usually when you're classifying, you just take whichever class has the least score, right? [Speaker:ANDREJ KARPATHY] Oh, yeah. So in practice, you usually also, when you're doing this classification, you add an extra— 2 things. One, you can add an extra class to say background and say like, oh, there's nothing here. Another thing you can do is not— is do actually multi-label classification. So you could output multiple positive things. That's actually pretty easy to do. It's just instead of a softmax loss, you have independent regression losses or independent logistic regressors per class. So that can actually let you say yes on multiple classes at one point. But that's just swapping out a loss function. So that's pretty easy to do. Right? So actually, like we said, a problem with this approach is that there's a whole bunch of different positions we need to evaluate. The solution, sort of, as of a couple years ago, was just use really fast classifiers and try them all. So actually, detection is this really old problem in computer vision. So you should probably have a little bit more historical perspective. So starting in about 2005, there was this really successful approach to— at the time, really successful— for pedestrian detection that used this feature representation called histogram of oriented gradients. So if you recall back to homework 1, you actually used this feature on the last part to actually do classification as well. So this was actually sort of the, the best feature that we had in computer vision circa around 2005. So the idea is we're just going to do linear classifiers on top of this feature, and that's going to be our classifier. So linear classifiers are really fast. So the way this works is that we compute our histogram of oriented gradient feature for the whole image at multiple scales. And we run this linear classifier at every scale, every position. Just do it really fast. Just do it everywhere because it's a linear classifier and it's fast to evaluate. And this worked really well in 2005. Sort of people took this idea and worked on it a little bit more for the next couple of years. So sort of one of the most important detection paradigms pre-deep learning was this thing called the deformable parts model. So I don't want to go too much into the details of this. But I mean, the basic idea is that we're still working on these histogram of orient gradient features. But now our model, rather than just being a linear classifier, we have this linear— this linear sort of template for the object. And we also have these templates for parts that are allowed to sort of vary over spatial positions and deform a little bit. And they have some fancy thing called a latent SVM to learn these things. And a really fancy dynamic programming algorithm to actually evaluate this thing really fast at test time. So this is actually kind of fun if you enjoy algorithms. This thing— this part is kind of fun to think about. But the end result is that it's a more powerful classifier because it allows a little bit of deformability in your model. And you can still evaluate it really fast. So we're still just going to evaluate it everywhere, every scale, every position, every aspect ratio. Just do it everywhere because it's fast. And this actually worked really well in 2010, around there. So that was sort of state of the art in detection for many problems at that time. So this is— I don't want to spend too much time on this, but there was a really cool paper last year that argued that these DPM models are actually just a certain type of ConvNet, right? So, right? Because these histograms of oriented gradients are like little edges. We can compute edges with convolutions. And then the histogram is kind of like pooling, that sort of thing. So if you're interested, check out this paper. It's kind of fun to think about. Right? But we really want to work on— make this thing work on classifiers that are not fast to evaluate, like maybe a CNN. So here, this problem is still hard, right? We have many different positions we want to try when we probably can't actually afford to try them all. So the solution is that we don't try them all. We have some other thing that sort of guesses where we want to look. and then we only apply our expensive classifier at those smaller number of locations. So that idea is called region proposals. So we— a region proposal method is this thing that takes in an image and then outputs a whole bunch of regions where maybe possibly an object might be located. So one way you can think about region proposals is that they're kind of like a really fast class-agnostic object detector. They don't care about the class. They're not very accurate, but they're pretty fast to run. And they give us a whole bunch of boxes. And the general intuition behind these region proposal methods is that they're kind of looking for blob-like structures in the image. So like objects are generally— like the dog, I mean, if you kind of squint, it looks kind of like a white blob. The cat looks kind of like a white blob. These flowers are kind of blobby. The eyes and the nose are kind of blobby. So when you run these region proposal methods, a lot of times what you'll see is they'll kind of put boxes around a lot of these blobby regions in the, in the image. So probably the most famous region proposal method is called Selective Search. You don't really need to know exact— in too much detail how this works, but the idea is that you start from your pixels and you kind of merge adjacent pixels together if they have similar color and texture. And form these connected blob-like regions. And then you merge up these blob-like regions to get bigger and bigger blobby parts. And then for each of these different scales, you could actually convert each of these blobby regions into a box by just drawing a box around it. So then by doing this over multiple scales, you end up with a whole bunch of boxes around sort of a lot of blobby stuff in the image. And it's reasonably fast to compute. It actually cuts down the search space quite a lot. But Selective Search really isn't the only game in town. It's just maybe the most famous. There's a whole bunch of different region proposal methods that people have developed. So there was this paper last year that actually did a really cool, thorough scientific evaluation of all these different region proposal methods and sort of gave you the pros and the cons of each and all that kind of stuff. But I mean, my takeaway from this paper was just use edge boxes if you have to pick one. 'cause it's really fast. You can run it in about a third of a second per image compared to about 10 seconds for selective search. But the more stars is better, and it gets a lot of stars. So it's good. [LAUGHTER] Right. So now that we have this idea of region proposals and we have this idea of a CNN classifier, let's just put everything all together. So that's— and so this idea was sort of first put together in a really nice way in 2014 in this method called R-CNN. So the idea is it's a region-based CNN method. So it's— I mean, it's pretty simple now that we've seen all the pieces. We have an input image. We're going to run a region proposal method like Selective Search to get about maybe 2,000 boxes of different scales and positions. I mean, 2,000 is still a lot, but it's a lot less than all possible boxes in the image. Now for each of those boxes, We're going to crop and warp that image region to some fixed size and then run it forward through a CNN to classify. And then this CNN is going to have a regression head and a regression head here and a classification head. They used SVMs here. So the idea is that this regression head can sort of correct for region proposals that were a little bit off. Right? So this actually works really well. It's really simple. Yeah, it's pretty cool. But unfortunately, the training pipeline is a little bit complicated. So the way that you end up training this RCNN model is like many models, you first start by downloading a model from the internet that works well for classification. Originally, they were using an AlexNet. Then next, we actually want to fine-tune this model for detection because this classification model was probably trained on ImageNet for 1,000 classes. But your detection dataset has a different number of classes, and the image statistics statistics are a little bit different. So you still run this— you'd still train this network for classification, but you have to add a couple new layers at the ends to deal with your classes and to help you deal with the slightly different statistics of your image data. So here, you're just doing classification still, but you're not running it on whole images. You're running it on positive and negative regions of your images from your detection dataset. Right, so you initialize a new layer, and you train this thing again for your dataset. Next, we actually want to cache all these features to disk. So for every image in your, in your dataset, you run selective search. You run that image, you extract those regions, you warp them, you run it through the CNN, and you cache those features to disk. And something important for this step is to have a large hard drive. The Pascal dataset is not too big, maybe order of a couple tens of thousands of images, but extracting these features actually takes hundreds of gigabytes. So that's not so great. And then next we have this— we want to train our SVMs to actually be able to classify our different classes based on these features. So here we want to run a bunch of— we want to train a bunch of different binary SVMs to classify image regions as to whether or not they contain or don't contain that one object. So this goes back to a question a little bit ago that sometimes you actually might want to have one region have multiple positive— be able to output yes on multiple classes for the same image region. And one way that they do that is just by training separate binary SVMs for each class. So then this is sort of an offline process. They just used LibSVM. So you have these features. These are maybe— those aren't positive samples for a cat SVM. That doesn't make any sense. But you get the idea. You have these different image regions. You have these features that you've saved to disk for those regions, and then you divide them into positive and negative samples for each class, and you just train these binary SVMs. So you do this for cat. You do this— the same thing for dog. And you just do this for every class in your dataset. Right, and then there's another step, right? So then there's this idea of box regression. So sometimes your region proposals aren't perfect. So what we actually want to do is be able to regress from these cached features to a correction onto the region proposal. And that correction has this kind of funny parameterized normalized representation that you can read details about in the paper. But kind of the intuition is that maybe for this region proposal, it was pretty good. We don't really need to make any corrections. But maybe this one in the middle, the proposal was too far to the left. It should have— the correct ground truth is a little bit to the right. So we want to regress to this correction vector that actually tells that we need to shift a little bit to the right. Or maybe this guy is a little bit too wide. It includes too much of the stuff outside the cat. So we want to regress to this correction vector that tells us we need to shrink the region proposal a little bit. So again, this is just a linear— they just do linear regression, which you know from 229. You have these features. You have these targets. You just train linear regression. And that's that. So before we look at the results, we should talk— I want to talk a little bit about the different datasets that people use for detection. There's kind of 3 that you'll see in practice. One is the Pascal VOC dataset. It was pretty important, I think, in the earlier 2000s. But now it's a little bit small. So this one's about— this one's 20 classes and about 20,000 images and tends to have about 2 objects per image. So because this is a relatively smallish dataset, you'll see a lot of detection papers work on this just because it's easier to handle. But there's also an ImageNet detection dataset. ImageNet runs a whole bunch of challenges, as you've probably seen by now. We saw classification. We saw localization. There's also an ImageNet detection challenge. But for detection, there's only 200 classes, not the 1,000 from classification. But it's, it's very big, almost half a million images. So you don't see as many papers work on it just because it's kind of annoying to handle. But there's only about one object per image. And then more recently, there's this one from Microsoft called COCO, which has fewer classes, fewer images, but actually has a lot more objects per image. So people like to work on that now because it's more interesting. Right. There's also this— when you're talking about detection, there's this funny evaluation metric we use called mean average precision. I don't really want to get too much into the details. What you really need to know is that it's a number between 0 and 100, and 100 is good. And it also— I mean, kind of the intuition is that you want to have the right— you want to— have true positives get high scores. And you also want to have some threshold on— that your boxes you produce need to be within some threshold of the correct box. And you can— usually this— that threshold is 0.5 on intersection over union. But you'll see different challenges use slightly different things for that. Right? So let's— now that we understand the datasets and the evaluation, let's see how R-CNN did. Right? So this is on the PASCAL— on two versions of the PASCAL dataset. Like I said, it's smaller, so you'll see a lot of results on this. There's different versions, one in 2007, 2010. You'll often see people use those just because the test data is publicly available. So it's easy to evaluate. Yeah, but so in this, this deformable parts model that we saw from 2011 from a couple slides ago is getting about 20, about 30 on mean average precision. There's this other method called Regionlets from 2013 that was sort of the state of the art that I could find right before deep learning. But it's sort of a similar flavor. You have these features, and it's classifiers on top of features. And R-CNN is this pretty simple thing we just saw and actually improves the performance quite a lot. So the first thing to see is we had a big improvement when we just switched this pretty simple framework using CNNs. And actually, this result here is without the bounding box regressions. This is only using the region proposals and the SVMs. Actually, if you include this additional bounding box proposal step, it actually helps quite a bit. Another fun thing to note is that if you take R-CNN and you do everything the same except you use VGG16 instead of AlexNet, you get another pretty big boost in performance. So this is kind of similar to what we've seen before, that just using these more powerful image features tends to help a lot of different tasks. Right, so this is really good, right? We've, we've done like a huge improvement on detection compared to 2013. That's amazing. But R-CNN is not perfect. It has some problems. Right, so it's pretty slow at test time, right? We saw that we have maybe 2,000 regions. We need to evaluate our CNN for each region. That's kind of slow. We also had this, this slightly subtle problem where our SVM and our regression, those were sort of trained offline using like libsvm and linear regression. So actually the weights of our, of our CNN didn't really have the chance to update in response to what those parts of the, of the network of those objectives wanted to do. And we also had this kind of complicated training pipeline that was a bit of a mess. So to fix these problems, a year later, we had this thing called Fast-R-CNN. So Fast-R-CNN was presented pretty recently at— in ICCV just in December. But the idea is really simple. We're just going to swap the order of extracting regions and running the CNN. So this is kind of related to this sliding window idea we saw with Overfeat. So here the pipeline at test time looks kind of similar. We have this input image. We're going to not— we're going to take this high-resolution input image and run it through the convolutional layers of our network. And now we're going to get this high-resolution convolutional feature map. And now our region proposals, we're going to extract directly features for those region proposals from this convolutional feature map using this thing called ROI pooling. And then the regions The features for these— the convolutional features for those regions will be fed into our fully connected layers and will again have a classification head and a regression head like we saw before. So this is really cool. It's pretty great. It solves a lot of the problems that we just saw with R-CNN. So R-CNN is really slow at test time. We solve this problem by just sharing this computation of convolutional features across different region proposals. R-CNN also had these problems at training time where we had this messy training pipeline. Had this problem where we were training different parts of the network separately. And the solution is pretty simple. We just train it all together all at once. Don't have this complicated pipeline, which we can actually do now that we have this pretty nice function from inputs to outputs. So you can see that Fast R-CNN actually solves quite a lot of the problems that we saw with vanilla R-CNN. Sort of the really interesting technical bit in Fast R-CNN was this problem of ROI, region of interest pooling. Pooling. So the idea is that we have this input image that's probably high resolution. And we have this region proposal that's maybe coming out of Selective Search or EdgeBoxes or something like that. And we can put this region— this high resolution image through our convolutional and pooling layers just fine because those are sort of scale invariant. They scale up to different sizes of inputs. But now the problem is that the fully connected layers from the— our pretrained network are expecting these pretty low-res conv features, whereas these features from the whole image are high-res. So now we solve this problem in a pretty straightforward way. So given this region proposal, we're going to project it onto sort of the spatial part of that conv feature volume. Now we're going to divide that conv feature volume into a little grid, right? Divide that thing into this H by W grid that the downstream layers are expecting. And we do max pooling within each of those grid cells. So now we've— through this pretty simple strategy, we've taken this region proposal and we've shared convolutional features. Now we've extracted this fixed-size output for that region proposal. So this is basically just swapping the order of convolution and warping and cropping. That's one way to think about it. And also, this is a pretty nice operation because since this thing is basically just max pooling and we know how to backpropagate through max pooling, pooling, you can backpropagate through these region of interest pooling layers just fine. And that's what really allows us to train this whole thing in a joint way. Right, so let's see some results. And these are actually pretty cool, pretty amazing, right? So for training time, R-CNN, it had this complicated pipeline. We had to save all this stuff to disk. We had to do all this stuff independently. And even on that pretty small Pascal dataset, it took 84 hours to train. Fast R-CNN is much faster. You can train it in a day. As far as test time, vanilla R-CNN is pretty slow because, again, we're running these independent forward passes of the CNN for each region proposal. Whereas for fast R-CNN, we can sort of share the computation between different region proposals and get this gigantic speedup at test time, 146. That's great. Amazing. And in terms of performance, I mean, it does a little bit better. It's not a drastic difference in performance. But this could probably be attributed to this fine-tuning property that with fast R-CNN, you can actually fine-tune all parts of the convolutional network jointly to help with these output tasks. And that's probably why you see a bit of an increase here. Right? So this is great, right? What's— what could possibly be wrong with fast R-CNN? It looks amazing. The big problem is that these test time speeds don't include region proposals. Right? So now fast R-CNN is so good that actually the bottleneck is computing region proposals. That's pretty cool. So once you factor in the speed of actually computing these region proposals on CPU, you can see that a lot of our speed benefits disappear, right? We're only 25x faster. And we kind of lost that beautiful 100x speedup. Also, now because it takes maybe 2 seconds to run actually per image end to end, you can't really use this real time. It's still kind of an offline processing thing. Right, so the solution of this should be pretty obvious, right? We're already using a convolutional network for regression. We're already using it for classification. Why not use it for region proposals too, right? This should work, maybe. Kind of crazy. So that's a paper. Anyone want to guess the name? Fast, fast RCNN? Yeah, so it's Faster RCNN. Yes, they were really creative here. But the idea is pretty simple. From fast R-CNN, we're already taking our input image and we're computing these big convolutional feature maps over the entire input image. So then instead of using some external method to compute region proposals, they add this little thing called a region proposal network that looks directly at these— that looks at these last layer convolutional features and is able to produce region proposals directly from that convolutional feature map. And then once you have region proposals, you just do the same thing as Fast R-CNN. You use this ROI pooling, and all the upstream stuff is the same as Fast R-CNN. So really the novel bit here is this region proposal network. It's really cool, right? We're doing the whole thing in one giant convolutional network. Right, so the way that this region proposal network works is that we're sort of— we receive as input this convolutional feature map that's maybe coming out of the last layer of our convolutional features. And we're going to add, like most things, our region proposal network is a convolutional network. So actually, this is a typo. This is a 3x3 conv. I'll fix that. So we have a sort of a sliding window approach over our convolutional feature map. But sliding window is just a convolution. So we just have a 3x3 convolution on top of this feature map. And then we have this familiar two-head structure inside the region proposal network. Where we're doing classification, where here we just want to say whether or not it's an object, and also regression to regress from this sort of position onto an actual region proposal. So the idea is that the position of the sliding window relative to the feature map sort of tells us where we are in the image, and then these regression outputs sort of give us corrections on top of this, this position in the feature map. but actually they make it a little bit more complicated than that. So instead of regressing directly from this position in the convolutional feature map, they have this notion of these different anchor boxes. So you could imagine taking these different sized and shaped anchor boxes and sort of pasting them in the original image at the point of the image corresponding to this point in the feature map, right? Like in Fast R-CNN, we were projecting forward from the image into the feature map. Now we're doing the opposite. We're projecting from the feature map back into the image for these anchor boxes. So then for each of these anchor boxes, they use sort of N convolutional anchor boxes. They use the same ones at every position in the image. And they— for each of these anchor boxes, they produce a score as to whether or not that anchor box corresponds to an object. And they also produce 4 regression coordinates that can correct that anchor box in similar ways that we saw before. Before. And now this region proposal network, you can just train to try to predict sort of as a class-agnostic object detector. So Faster R-CNN, in the original paper, they train this thing in kind of a funny way where first they train the region proposal network, then they train Fast R-CNN, then they do some magic to merge together. And at the end of the day, they have one network that produces everything. So this, this is a little bit messy. But in the original paper, they described this thing. But since then, they've had some unpublished work where they actually just train the whole thing jointly, where they're sort of— they have one big network where you have an image coming in. You have this— inside the region proposal network, you have a classification loss to classify whether each region proposal is or is not an object. You have these bounding box regressions inside the region proposal network to regress on top of your convolutional anchors. And then from fast— then we do ROI pooling and do this fast RCNN trick. And then at the end of the network, we have this classification loss to say which class it is and this regression loss to correct— to have a correction on top of the region proposal. So this is this big thing. It's just one big network with 4 losses. Yeah. Just for clarification, you're saying that the proposal works on a 3-bit 3x3 convolution. So the proposal scores and regression coordinates are produced by a 3x3, a 3x3 and then a pair of 1x1 convolutions off the feature map. So the idea is that we're looking at these different anchor boxes of different positions and scales, but we're actually looking at the same position in the feature map to classify those different anchor boxes. But you'd have different— you learn different weights for the different anchors. Why is it 3 by 3? How does that relate to the whole feature size? I think it's mostly empirical. So the idea is just you want to have a little bit of nonlinearity. You could imagine just doing sort of a direct 1 by 1 convolution directly off the feature maps. But I think they don't discuss this in the paper, but I'm guessing just a 3 by 3 tends to work a bit better. But there's no really deep reason why you do that. It could be more. It could be less. It could be a bigger kernel. You have this little convolutional network with two heads. That's the main point. Any other questions? Yeah. So we were taking that map at the— like after doing all the convolution layers, right? Mm-hmm. So what happens if we just prune the image just to that particular area before we start doing all the convolution and just keep all the neighboring pixels? Wouldn't it be like sort of faster? Sorry, I don't quite understand because your convolutional feature map corresponds to the whole image. The point is that we don't actually want to process the whole image. We want to pick out some regions of the image to do more processing on, but we need to choose those regions somehow. So I'm saying like what if you just select those regions like before doing all the previous convolution there? Yeah, so that's basically this idea of using external region proposals. So when you do external region proposals, you're sort of picking it first before you do the convolutions. But it's just sort of a nice thing if you can do it all at once. So convolutions are kind of this general, like really general processing that you can do to the image. So you're kind of hoping that convolutions are good enough for classification, they're good enough for regression. The types of information that you have in those convolutions is probably good enough for classifying regions as well. As well. So it's actually a computational savings because at the end of the day, you end up using that same convolutional feature map for everything, for the region proposals, for the downstream classification, for the downstream regression. So that's actually why you get the speedup here. Yeah, but good question. Yeah, so then we have this big network. We train it with 4 losses. And now we can do object detection sort of all at once. That's pretty cool. So if we look at results comparing the 3 RCNNs of various velocities, then we— for original RCNN, it took about 50 seconds at test time per image. This is counting the region proposals. This is counting running the CNN separately for each region proposal. So that's pretty slow. Now fast RCNN, we saw it was sort of bottlenecked by the region proposal time. But once we move to faster RCNN, then those region proposals are basically coming for free since they're just— the way we compute region proposals is just a tiny 3x3 convolution and a couple 1x1 convolutions. So those are very cheap to evaluate. So then at test time, faster R-CNN runs in a fifth of a second on a pretty high-resolution image. So that's actually— yeah? Are there any issues with objects close to the edge of the image? Like, because effectively— [Speaker:AUDIENCE] So basically when you're padding or whatever, you're going to end up— you don't gain information from those at the edge of the image. [Speaker:ANDREJ KARPATHY] Well, I mean, all your convolutions always have zero padding. So you're not— one of the ideas behind zero padding is you're hoping not to throw away information from the edges. So I think maybe you might have a problem with— if you didn't do the zero padding, it might be more of a problem. But I mean, as we've sort of discussed before, the fact that you're adding that zero padding might affect the statistics of those features. It could maybe be a bit of a problem. But in practice, it seems to work just fine. But actually, that type of analysis of where do we have failure cases, where do we get things wrong, is a really important process when you're developing new algorithms. And that can give you insight onto what might make things better. Yeah, any other questions? Yeah. [Speaker:AUDIENCE] If I'm only interested actually in the classification task, maybe even with multiple objects in the same image, Um, do these localization methods still, um, actually help me to, to boost my, my classification accuracies or scores? Or can I say like, actually it's not really important then for me if I'm only interested in the classification itself? So I mean, I haven't— I have the intuition that maybe it might help, but it's actually kind of hard to do that experiment because the datasets are different, right? Because when you work on a classification dataset like ImageNet, that's one thing, but then when you work on detection, it's this other dataset. And I haven't— you could imagine trying to classify the detection images based on what objects are present, but I haven't really seen any really good comparisons that try to study that empirically. But I mean, that'd be a cool experiment to have in your project. Yeah. [Speaker:AUDIENCE] So if, for example, you wanted to change your bounding boxes so that they also take rotation into account, would you just add a theta parameter? The bounding boxes, like their representation, and then run it through this network? And would that work in a similar way? So that's actually a very good question. So then you have this problem with ROI pooling, right? Because the way that the ROI pooling worked was by dividing that thing into this fixed grid and doing max pooling. Once you do rotations, it's actually kind of difficult. There's this really cool paper from DeepMind over the summer called Spatial Transformer Networks. That actually introduces a really cool way to solve this problem. And the idea is that instead of doing ROI pooling, we're going to do bilinear interpolation, kind of like you might use for textures in graphics. So once you do bilinear interpolation, then you actually can do maybe these crazy regions. So yeah, that's definitely something people are thinking about, but it hasn't been incorporated into the whole pipeline yet. Yeah? Couldn't you alternatively just train on all sorts of rotations of your dataset and just learn the different rotations of objects you've had to— You could. That would be slow. Then you're back in this sort of RCNN regime, right? And look at that, 250 times slower. Do you really want to pay that price? I mean, I think another practical concern with rotated objects is that we don't really have that ground truth in our datasets. So for most of these detection datasets, the only ground truth information we have are these axis-aligned bounding boxes. So it's hard. You don't really have a ground truth position. So that's kind of a practical concern why I think people haven't really explored this so much. Yeah, so the end story with faster R-CNN is it's super fast, and it does about the same. Right? So that's good. It works. So actually, really interesting is now at this point, you can actually understand the state of the art in object detection. So this is sort of the best object detector in the world. It crushed everyone at the ImageNet— ImageNet and COCO challenges in December. And like most other things, it's this deep residual network. So the best object detector in the world right now is 101-layer residual network plus faster RCNN plus a couple other goodies here. So we talked about faster RCNN. We saw ResNet last year. They have— to get an extra— always for competitions, you need to add a couple of crazy things to just get a little bit boost in performance. So here, this box refinement, they actually do multiple steps of refining the bounding box because you, you sort of saw that in the fast RCNN framework, you're doing this correction on top of your region proposal. So then you could actually feed that back into the network and reclassify and re-get another prediction. So that's this box refinement step. It gives you a little bit of boost. They add context. So in addition to classifying just the, just the region, They get a vector that gives you the whole features for the entire image that sort of gives you more context than just that little crop. And that gives you a little bit more performance. And they also do multiscale testing, kind of like we saw in Overfeat. So they actually run the thing on images at different sizes at test time and then aggregate over those different sizes. And when you put all those things together, you win a lot of competitions. So this thing— One on— so COCO actually— Microsoft COCO actually runs a detection challenge, and they won the detection challenge on COCO. We can also look at the rapid progress on the ImageNet detection challenges over the last couple years. So you can see in 2013 was sort of the first time that we had these deep learning detection models. So OverFeat that we saw for localization, they actually submitted— they had a version of their system that works on detection as well. By sort of changing the logic by which they merge bounding boxes. And they did pretty good, but they were actually outperformed by this other, this other group called U-Vision that was sort of not a deep learning approach. They used a lot of features. But then in 2014, we actually saw both of these were deep learning approaches, and Google actually won that one by using a GoogleNet plus some other detection stuff on top of GoogleNet. And then in 2015, things went crazy, and these residual networks plus faster RCNN just crushed everything. So I think detection, especially over the last couple of years, has been a really exciting thing because we've seen this really rapid progress over the last couple of years in detection, like most other things. And another point I think is kind of fun to make is that actually, for all— like, again, to win competitions, you know, Andrej Karpathy said you ensemble and get 2%. So you always win competitions with an ensemble. But actually, sort of for fun, Microsoft also submitted their best single ResNet model So this was not an ensemble. And just a single ResNet model actually beat all the other things from all the other years. So that's actually pretty cool. Yeah, so that's the best detector out there. So this is kind of a fun thing, right? So this is a really— so we talked about this idea of localization as regression. So this funny thing called YOLO, you only look once, actually tries to pose the detection problem directly as a regression problem. So the idea is that we actually are going to take our input image and we're going to divide it into some spatial grid. They used a 7 by 7. And then within each element of that spatial grid, we're going to make a fixed number of bounding box predictions. They use b equals 2, I think, in most of their experiments. So then within each grid, you're going to predict maybe 2 b bounding boxes. So that's 4 numbers. You're also going to predict a single score for how much you believe that bounding box. And you're also going to predict a classification score for each class in your dataset. So then you can sort of take this, this detection problem and it ends up being regression where your input is an image and your output is this maybe 7 by 7 by 5B plus C tensor, right? So just a regression problem and just train it end-to-end with a CNN. So that's pretty cool. And it's sort of a neat approach. It's a little bit different than these region proposal things that we've seen before. Of course, sort of a problem with this is that there's an upper bound in the number of outputs that your model can have. So that might be a problem if your testing data has many, many more ground truth boxes than your training data. So this, this YOLO detector actually is really fast. It's actually faster than faster R-CNN, which is pretty crazy. But unfortunately, it tends to work a little bit worse. So they have this other thing called FastYOLO that I don't want to talk about. But— right? But so these are mean AP numbers on one of the Pascal datasets that we saw. So you can see YOLO actually gets 64. That's pretty good and runs at 45 frames per second. So this is obviously on a powerful GPU. But still, that's pretty much real time. That's amazing. There's also— I don't want to talk about that. Right, so then there's these different versions of fast and faster R-CNNs. You can see that these actually pretty much all beat YOLO in terms of performance but are quite a bit slower. Yeah, so that's actually kind of a neat twist on the detection problem. Yeah, so actually all these different detection metric— all these different detection models that we talked about today, they all pretty much have code up. They're released. You should maybe consider using them for your projects. Probably don't use RCNN. It's too slow. Fast RCNN is pretty good, but it requires MATLAB. Faster RCNN— there is actually a version of Faster RCNN that doesn't require MATLAB. It's just Python and Caffe. I haven't personally used it, but it's something you might want to try to use for your projects. I'm not sure how difficult it is to get running. And YOLO is actually, I think, maybe a good choice for some of you for projects because it's so fast that It might be easier to work with if you have not these really big powerful GPUs. And they actually have code up as well. Yeah, so that's actually— I got through things a little bit faster than I expected. So is there any questions on detection? Nope. Oh, yeah, question? So how big are each of these models? And then since you mentioned GPU, how big is each model? Yeah, so in terms of model size, it's pretty much— it's about the same as a classification model because when you're running on bigger image, especially for a faster RCNN, right? Because your convolutions, you don't really introduce any more parameters. The fully connected layers are not really any more parameters. You have a couple extra parameters for the region proposal network, but it's basically the same number of parameters as a classification model. All right. I guess we're done a little early today.