C4W2L11 State of Computer Vision

Deep learning has been successfully applied to computer vision. Natural language processing, speech recognition, online advertising, logistics, Many, many, many problems. There are a few things that are unique about the application of deep learning to computer vision, about the status of computer vision. In this video, I'm going to share with you some of my observations about deep learning for computer vision in the hope that that will help you better navigate the literature and the set of ideas out there and, and how you build these systems yourself for computer vision. So you can think of most machine learning problems as falling somewhere on the spectrum between where you have relatively little data to where you have lots of data. So, for example, I think that today we have a decent amount of data for speech recognition, at least relative to the complexity of the problem. And even though there are reasonably large data sets today for image recognition or image classification, because image recognition is just a complicated problem to look at all those pixels and figure out what it is, it feels like even though the online data sets are quite big, you know, like over a million images, feels like we still wish we had more data. And there's some problems like object detection, where we have even less data. So just as a reminder, image recognition was a problem of looking at a picture and telling you is this a cat or not? Whereas object detection is looking at a picture and actually you're putting the bounding boxes or telling you where in the picture the object, such as the cause are as well. And so because of the cost of getting the bounding boxes, um, it's just more expensive to label the objects and the boundary boxes. So we tend to have less data for object detection than for image recognition. And object detection is something we'll discuss next week. So if you look across a broad spectrum of machine learning problems, you see on average that when you have a lot of data, you tend to find people getting away with using simpler algorithms as well as less hand engineering. So there's just less needing to carefully design features for the problem. But instead you can have a giant neural network, even a simpler architecture, and have a neural network just learn whatever it wants to learn when you have a lot of data. Whereas in contrast, when you don't have that much data, then on average you see people engaging in more hand engineering. And if you want to be ungenerous, you know, you can say there are more hacks, But I think when you don't have much data, then hand engineering is actually the best way to get good performance. So when I look at machine learning applications, I think usually we have the learning algorithm has two Sources of knowledge. One source of knowledge is the label data, really the X, Y pairs you use for supervised learning. And the second source of knowledge is the hand engineering. And there are lots of ways to hand engineer a system. It can be from carefully hand designing the features to carefully hand designing the network architectures, to maybe other components of your system. And so when you don't have much labeled data, you just have to count more on hand engineering. And so I think computer vision is trying to learn a really complex function. And it often feels like we don't have enough data for computer vision. Even though data sets are getting bigger and bigger, often we just don't have as much data as we need. And this is why the state of computer vision historically and even today has relied more on hand engineering. And I think this is also why the, the field of computer vision has developed rather complex network architectures is because in the absence of more data, the way to get good performance is to spend more time architecting or, you know, fiddling around with the network architecture. And in case you think I'm being derogatory of hand engineering, that's not at all my intent. When you don't have enough data, hand engineering is a very difficult, very skillful tool task that requires a lot of insight. And someone that is insightful at hand engineering will get better performance and is a great contribution to a project to do that. Hand engineering. When you don't have enough data, it's just when you have lots of data, then I wouldn't spend time hand engineering, I would spend time, you know, building up the learning system instead. But I think historically the field of computer vision has used very small data sets. And so historically the computer vision literature has relied on a lot of hand engineering. And even though in the last few years the amount of data we throw at computer vision task has increased dramatically, I think that that has resulted in a significant reduction in the amount of hand engineering that's being done. But there's still a lot of hand engineering of network architectures in computer vision, which is why you see very complicated hyper parameter choices in computer vision more complex than you do in a lot of other disciplines. And in fact, because you usually have smaller object detection data sets than image recognition data sets, when we talk about object detection, that is, tasks like this next week, you see that the algorithms, or if you want to be, you see that the algorithms become even more complex and has even more, you know, specialized components. Fortunately, one thing that helps a lot when you have little data is transfer learning. And I would say for the example from the previous slide of the Tigger msti, neither detection problem, you have so little data that transfer learning, you know, will help a lot. And so that's another set of techniques that's used a lot for when you have relatively little data. If you look at the computer vision literature and look at the set of ideas out there, you also find that people are really enthusiastic. They're really into doing well on standardized benchmark data sets and on winning competitions. And for computer vision researchers, if you do well on a benchmark, it's easier to get a paper published. So there's just a lot of attention on doing well on, on these benchmarks. And the positive side of this is that it helps the whole community figure out what are the most effective algorithms. But you also see in the papers people do things that allow you to do well on a benchmark, but that you wouldn't really use in a production or a system that you deploy in an actual application. So here are a few tips for doing well on benchmarks. These are things that I don't know myself pretty much ever use if I'm putting a system into production that is to actually serve customers. But one is ensembling. And what that means is after you've figured out what neural network you want, train several neural networks independently and average their outputs. So initialize, say, three or five or seven neural networks randomly and train up all of these neural networks and then average their outputs. And by the way, it's important to average their outputs. Y hat don't average their weights. That won't work. Look at your, say, seven neural networks. They'll have seven different predictions and average that. And this will cause you to do maybe 1% better or 2% better or. So it's a little bit better on some benchmark and this will cause you to do a little bit better, you know, maybe sometimes as much as 1 or 2%, which can really help win a competition. But because ensembling means that to test on each image, you might need to run an image through anywhere from, say, three to 15 different networks. Quite typical. This slows down your running time by a factor of three to 15 or sometimes even more. And so ensembling is one of those tips that people use for doing well in benchmarks and for winning competitions. But that, I think is almost never used in production to serve actual customers, I guess, unless you have a huge computational budget and don't mind burning a lot more of it per customer image. Another thing you see in papers that really helps on benchmarks is multi crop and test time. So what I mean by that is you've seen how you can do data augmentation and multi crop is a form of applying data augmentation to your test image as well. So for example, let's say you have a cat image and just copy it four times here, including two mirror versions. There's a technique called the 10 crop which basically says let's say you take this central region, that crop, and run it through your classifier and then take that prop, upper left hand corner run through your classifier, upper right hand corner shown in green, lower left shown in yellow, lower right shown in orange, and run that through your classifier and then do the same thing with the mirrored image. Right? So take the central crop, then take the four corners crops. So that's one central crop here and here. There's four corners crop here and here. And if you add these up, that's 10 different crops of the image. So hence the name 10 crop. And so what you do is you run these 10 images so through your classifier and then average the results. So if you have the computational budget, you could do this. Maybe you don't need as many as 10 crops, you can use a few crops and this might get you a little bit better performance in a production system. By production I mean a system you're deploying for actual users. But this is another technique that is used much more for doing well on benchmarks than in actual production system. And one of the big problems of ensembling is that you need to keep all these different networks around. And so that just takes up a lot more computer memory for multi crop. I guess at least you keep just one network around so it doesn't suck up as much memory, but it still slows down your runtime quite a bit. So these are tips you see, and research papers will refer to these tips as well. But I personally do not tend to use these methods when building production systems, even though they are great for doing better on benchmarks and on winning competitions. Because a lot of computer vision problems are in a small data regime, others have done a lot of hand engineering of the network architectures and a neural network that works well on one vision problem, often, maybe surprisingly, but it just often will work on other vision problems as well. So to build a practical system, often you do well starting off with someone else's new network architecture. And you can use an open source implementation if possible, because the open source implementation might have figured out all the finicky details like the learning rate, decay schedule and other hyperparameters. And finally, someone else might have spent weeks training a model on half a dozen GPUs and on over a million images. And so, by using someone else's pre trained model and fine tuning on your data set, you can often get going much faster on an application. But of course, if you have the compute resources and the inclination, don't let me stop you from training your own networks from scratch. And in fact, if you want to invent your own computer vision algorithm, that's what that's what you might have to do. So that's it for this week. I hope that seeing a number of computer vision architectures helps you get a sense of what works. In this week's programming exercises, you actually learn another programming framework and use that to implement resonance. So I hope you enjoy that programming exercise, and I look forward to seeing you next week.