Neural networks can predict X,Y coordinates for specific landmarks on objects, not just bounding boxes. A 64-landmark face model needs 129 output units. Consistent label ordering and labeled training data are essential for applications like emotion recognition and AR filters.
Questions this video answers
129 output units (1 binary face indicator + 64 landmarks × 2 coordinates = 1 + 128).
Each landmark index must be consistent across images (e.g., Landmark 1 always refers to the same anatomical point).
Chapters
In the previous video, you saw how you can get a neural network to output four numbers, BX by BH and bw to specify the bounding box of an object you want a neural network to localize. In more general cases, you can have a neural network just output X and Y coordinates of important points in image, sometimes called landmarks, that you want the neural network to recognize. Let me show you a few examples. Let's say you're building a face recognition application and for some reason you want the algorithm to tell you where is the corner of someone's eye. So that point has an X and Y coordinate. So you can just have a neural network have its final layer and have it just output two more numbers, which I'm going to call LX and ly to just tell you the coordinates of that corner of the person's eye. Now, what if you wanted to tell you all four corners of the eye really, of both eyes? So if we call the points the first, second, third and fourth points, going from left to right, then you could modify the neural network now to output L1x, L1y for the first point and L2x, L2y for the second point, and so on. So, so that the neural network can output the estimated position of all those four points of the person's face. But what if you don't want just those four points? What do you want to output? This point and this point and this point and this point, you know, along the eye, maybe output some key points along the mouth so you can extract the mouth shape and tell if the person is smiling or frowning. Maybe extract a few key points along the edges of the nose. But you could define some number. For the sake of argument, let's say 64 points or 64 landmarks on the face. Um, maybe even some points, you know, that help you define. The edge of the face defines a draw line. But by selecting a number of landmarks and generating a label training set that contains all of these landmarks, you can then have the neural network tell you which where are all the key positions or the key landmarks on a face. So what you do is you have this image person's face as input, have it go through a convnet and have a convnet, then have some set of features, maybe have it output 0 or 1, like is there a face in this or not? And then have it also output L1X, L1Y, and so on down to L64X, L64Y. And here I'm using L to stand for a landmark. So this example would have 129 output units. One for is there a face or not? And then if you have 64 landmarks, that's 64 times two, so 128 plus one output units. And this can take tell you if there's a face as well as where all the key landmarks on the face. So you know, this is a basic building block for recognizing emotions from faces. And if you played with the Snapchat and the other entertainment, you know, sort of AR augmented reality filters, like the Snapchat filters, can I draw a crown on the face and have other special effects? Being able to detect these landmarks on a face is also a chance key building block for the computer graphics effects that warp the face or draw on various special effects like put a crown or a hat on a person. Of course, in order to train a network like this, you will need a label training set. We have a set of images as well as labels Y where people, where someone will have had to go through and laboriously annotate all of these landmarks. One last example, if you are interested in people pose detection, you could also define a few key positions like the midpoint of the chest, the sh, the left shoulder, left elbow, the wrist, and so on. And just have a neural network, you know, sort of annotate key positions in the person's pose as well. And by having a neural network output all of those points that I'm annotating, you could also have the neural network output the pose of the person. And of course to do that you also need to specify these key landmarks like maybe L1x and L1y is the midpoint of the chest down to maybe O32x L32y if you use 32 coordinates to specify the pose of the person. So this idea might seem quite simple of just adding a bunch of output units to output the XY coordinates of different landmarks you want to recognize. To be clear, the identity of Landmark 1 must be consistent across different images. Like maybe Landmark 1 is always this corner of the eye, Landmark 2 is always this corner of the Eye, Landmark 3 Landmark 4 and so on. So the labels have to be consistent across different images. But if you can hire labelers or label yourself a big enough data set to do this, then a neural network can output, you know, all of these landmarks which you can then use to carry out other interesting effects such as estimated pose of a person, maybe try to recognize someone's emotion from a picture and and so on. So that's it for landmark detection. Next, let's take these building blocks and use it to start building up towards object detection.
