Whether to Use End-To-End Deep Learning (C3W2L10)

Let's say you're building a machine learning system and you're trying to decide whether or not to use an end to end approach. Let's take a look at some of the pros and cons of end to end deep learning so that you can come away with some guidelines on whether or not an end to end approach seems promising for your application. Here are some of the benefits of applying end to end learning. First, is that end to end learning really just lets the data speak. So if you have enough x comma y data, then whatever is the most appropriate function mapping from X to Y. If you train a big enough neural network, hopefully the neural network will figure it out. And by having a pure machine learning approach, your neural network learning an input from X and Y may be more able to capture whatever statistics are in the data rather than being forced to reflect human preconceptions. So, for example, in the case of speech recognition, earlier speech systems had this notion of a phoneme which was a basic unit of sound, like k, A and T for the word cat. And I think that phonemes are an artifact created by human linguists. I actually think that phonemes are a fantasy of linguists that are a reasonable description of language. But it's not obvious that you want to force your learning algorithm to think in phonemes. So. And if you let your learning algorithm learn whatever representation it wants to learn, rather than forcing your learning algorithm to use phonemes as a representation, then its overall performance might end up being better. The second benefit of end to end deep learning is that there's less hand designing of components needed. And so this could also simplify your design workflow that you just don't need to spend a lot of time hand designing features, hand designing these intermediate representations. How about the disadvantages? Here are some of the cons. First, it may need a large amount of data. So to learn this X to Y mapping directly, you might need a lot of data of X comma y. And we've seen in the previous video some examples of where you could obtain a lot of data for subtasks such as for face recognition. We could find a lot of data for finding a face in the image as well as identifying the face once you found the face. But there was just less data available for the entire end to end task. So you know X this is the input end of the end to end learning and Y is the output end. And so you need a lot of data XY with both the input end and the output end in order to train some of these Systems. And this is why we call it end to end learning as well, because you're learning a direct mapping from one end of the system, you know, all the way to the other end of the system. The other disadvantage is that it excludes potentially useful hand designed components. So machine learning researchers tend to speak disparagingly of hand designing things. But if you don't have a lot of data, then your learning algorithm doesn't have that much insight it can gain from your data if your training set is small. And so hand designing a component can really be a way for you to inject manual knowledge into the algorithm. And that's not always a bad thing. I think of a learning algorithm as having two main sources of knowledge. One is the data and the other is whatever you hand design, be it components or features or other things. And so when you have a ton of data, it's less important to hand design things. But when you don't have much data, then having a carefully hand designed system can actually allow humans to inject a lot of knowledge about a problem into a, into an algorithm. And that can actually be very helpful. So one of the downsides of end to end deep learning is that it excludes potentially useful hand design components. And hand design components could be very helpful if well designed. They could also be harmful if it really limits your performance, such as if you force an algorithm to think in phonemes when maybe it could have discovered a better representation by itself. So it's kind of a double edged sword that could hurt or help, but it does tend to help more. Hand design components tend to help more when you training on a small training set. So if you're building a new machine learning system and you're trying to decide whether or not to use end to end deep learning, I think the key question is do you have sufficient data to learn a function of the complexity needed to map from X to Y? I don't have a formal definition of this phrase complexity needed, but intuitively, if you're trying to learn a function from X to Y, that is looking at an image like this and recognizing the position of the bones in this image, then maybe this seems like a relatively simple problem to identify the bones in the image. And maybe you don't need that much data for that task, or given a picture of a person, maybe finding the face of that person in the image doesn't seem like that hard a problem. So maybe you don't need too much data to find the face of a person, or at least maybe you can find enough data to solve that task. Whereas in contrast, the function needed to look at a hand and map that directly to the age of a child. That seems like a much more complex problem that intuitively maybe you need more data to learn if you were to apply a pure end to end deep learning approach. So let me finish this video with a more complex example. Um, you might know that I've been spending time helping out an autonomous driving company drive AI. So I'm actually very excited about autonomous driving. So how do you build a car that drives itself? Well, here's one thing you could do, and this is not an end to end deep learning approach. You can take as input an image of what's in front of your car, maybe radar, lidar, other sensor readings as well. But you know, to simplify the description, just say you take a picture of what's in front and what's around your car. And then to drive your car safely, you need to detect other cars and you also need to detect pedestrians. You need to detect other things, of course, but we'll just present a simplified example here. Having figured out where are the other cars and pedestrians, you then need to plan your own route. So in other words, if you see where the other cars where the pedestrians, you need to decide how to steer your own car, what path to steer your own car for the next several seconds, and then having decided that you're going to drive a certain path, maybe if this is a top down view of a road, you know, that's your car, maybe you've decided to drive that path. That's what the path of what a route is. Then you need to execute this by generating the appropriate steering as well as acceleration and braking commands. So in going from your image or your sensor inputs to detecting cars and pedestrians, that can be done pretty well using deep learning. But then having figured out where the other cars and pedestrians are going to, you know, select this route to exactly how you want to move your car. Usually that's not done with deep learning. Instead that's done with a piece of software called motion planning. And if you ever take a class in robotics, you learn about motion planning. And then having decided what's the path you want to steer your car through, there'll be some other algorithm. I'm gonna say it's a control algorithm that then generates the exact decision that then decides exactly how much to turn the steering wheel and how much to step on the accelerator or step on the brake. So I think what this example illustrates is that you want to use machine learning or use deep learning to learn some individual components. And when applying supervised learning, you should carefully choose what types of XY mappings you want to learn, depending on what task you can get data for. And in contrast, it is exciting to talk about a pure end to end deep learning approach where you input an image and directly output a steering. But given data availability and the types of things we could learn with neural networks today, this is actually not the most promising approach or this is not an approach that I think teams have gotten to work best. And I think this is actually this pure end to end deep learning approach is actually less promising than most sophisticated approaches like this, given the availability of data and our ability to train neural networks today. So that's it for end to end deep learning. It can sometimes work really well, but you also have to be mindful of where you apply end to end deep learning. Finally, thank you and congrats on making it this far with me. If you finished last week's videos and this week's videos, then I think you'll already be much smarter and much more strategic and much more able to make good prioritization decisions in terms of how to move forward on your machine learning project, even compared to a lot of machine learning engineers and researchers that I see here in Silicon Valley. So congrats on all you've learned so far and I hope you now also take a look at this week's homework problems, which will give you an another opportunity to practice these ideas and make sure that you're mastering them.