Andrew Ng explains YOLO's object detection pipeline: dividing an image into an SxS grid with multiple anchor boxes per cell, training the network to predict objectness scores and bounding box coordinates, then filtering results with thresholding and non-max suppression. Real models use larger grids like 19x19 with 5 anchors each.
You've already seen most of the components of object detection in this video. Let's put all the components together to form the YOLO object detection algorithm. First, let's see how you construct your training set. Suppose you're trying to train an algorithm to detect three objects, pedestrians, cars and motorcycles. And you won't need to explicitly have the fourth background class. So just VCAS labels here. If you're using two anchor boxes, then the outputs y will be 3 by 3 because you're using a 3 by 3 grid cell by 2, this is the number of anchors by 8 because that's the dimension of this 8 is actually 5, which is and then plus the number of classes, so 5 because you have PC and then the bounding boxes, that's 5 and then C1, C2, C3. That dimension is equal to the number of classes. And you can either view this as 3x3x2x8 or 3x3x16. So to construct a training set, you go through each of these nine grid cells and form the appropriate target vector Y. So take this first grid cell. There's nothing worth detecting in that grid cell. None of the three classes, pedestrian, car, motorcycle, appear in the upper left grid cell. And so the target Y corresponding to that grid cell will be equal to this where PC for the first anchor box is zero because there's nothing associated with the first anchor box and is also zero for the second anchor box. And so all of these other values are don't cares. Now most of the grid cells have nothing in them, but for that box over there, you would have this target vector Y. So assuming that your training set has a bounding box like this for the car, it's just a little bit wider than it is tall. And so if your anchor boxes are that this is anchor box one, this is anchor box two, then the red box has just slightly higher IOU with anchor box two. And so the car gets associated with this lower portion of the vector. So notice then that PC associative anchor box one is zero. So you have don't care as all of these components, then you have this PC is equal to one, then you should use these to specify the position of the red bounding box and then specify that the correct object is plus two, right, that it is a car. So you go through this and for each of your nine grid positions, for each of your three by three grid positions, you would come up with a vector like this, come up with a 16 dimensional vector. And so that's why the final output volume is going to be 3 by 3 by 16. Oh, and as usual, for simplicity on this slide I've used a 3 by 3 grid. In practice it might be more like a 19 by 19 by 16. Or in fact if you use more anchor boxes, maybe 19 by 19 by 5 times 8, because 5 times 8 is 40, so it'd be 19 by 19 by 40. That's if you use five anchor boxes. So that's training. And you train a convnet that inputs an image, maybe 100 by 100 by 3. And your convnet would then finally output this output volume in our example 3 by 3 by 16 or 3 by 3 by 2 by 8. Next, let's look at how your algorithm can make predictions. Given an image, your neural network will output this 3 by 3 by 2 by 8 volume where for each of the nine grid cells you get a vector like that. So for the grid cell here, here on the upper left, if there's no object there, hopefully your neural Network will output 0 here and 0 here and it outputs some of the values. Your neural network can't output a question mark, it can't output a don't care. So output some numbers for the rest. But these numbers will basically be ignored because the neural network is telling you that there's no object there. So it doesn't really matter whether outputs is a bounding box or is a class. It'll basically just be some set of numbers, more or less noise. In contrast for this box over here, hopefully the value of y to the output for that box at the bottom left, hopefully it'll be something like zero for bounding box one. And then, you know, just output a bunch of numbers, just noise. Hopefully it'll also output a set of numbers that corresponds to specifying a pretty accurate bounding box for the car. So that's how the neural network will make predictions. Finally, you would run this through non max suppression. So just to make it interesting, let's look at a new test set image. Here's how you would run non max suppression. If you're using two anchor boxes, then for each of the nine grid cells you get two predicted bounding boxes. Some of them will have very low probability of very low PC, but you still get two predicted bounding boxes for each of the nine grid cells. So let's say those are the bounding boxes you get. And notice that some of the bounding boxes can go outside the height and width of the grid cell that it came from. Next you then get rid of the low probability predictions. So get rid of the ones that even the neural network says, gee, this object probably isn't there, so get rid of those. And then finally, if you have three classes you're trying to detect, if you're trying to detect pedestrians, cars and motorcycles, what you do is for each of the three classes independently, run non max suppression for the objects that were predicted to come from that class. But use non max suppression for predictions of the prediction class. Run non max suppression for the car class and run non max suppression for the most cycle class. But run that, you know, basically three times to generate the final predictions. And so the output of this is hopefully that you will have detected all the cars and all the pedestrians in this image. So that's it for the YOLO object detection algorithm, which is really one of the most effective object detection algorithms that also encompasses many of the best ideas across the entire computer vision literature that relate to object detection. And you get a chance to practice implementing many components of this yourself in this week's program exercise. So I hope you enjoy this week's program exercise. There's also an optional video that follows this one, which you can either watch or not watch as you please, but either way, I also look forward to seeing you next week.
