Nuts and Bolts of Applying Deep Learning (Andrew Ng)

So, you know, when we're organizing this workshop, my, my co organizers initially asked me, hey Andrew, end of the first day, go give a visionary talk. So until several hours ago, my talk was advertised as visionary talk. Um, but until, but, but I was preparing for this presentation over the last several days. Um, I tried to think what, what would be the most useful information to you, um, and what are the things that, you know, you could take back to work on Monday and, and do something different at your job next Monday. And I thought that metasecontext right now, as, as Peter mentioned, I lead Baidu's AI team. So it's a team of about a thousand people working on vision, speech, nlp, you know, lots of applications and machine learning. And so what I thought I'd do instead is instead of taking the shiniest pieces of deep learning that I know, I want to take the lessons that I saw at Baidu that are common to so many different academic areas, as well as applications, autonomous cars, augmented reality, advertising, web search, medical diagnosis. We take what I saw, the common lessons, the simple, powerful ideas that I've seen help drive a lot of machine learning progress at Baidu. And I thought, I will share those ideas with you because the patterns I see across a lot of projects I thought might be the patterns that would be most useful to you as well, whatever you are working on in the next several weeks or months. Um, so one common theme that will appear in, in this presentation today is that the workflow of organizing machine learning projects feels like parts of it are changing in the era of deep learning. So for example, one of the ideas I'll talk about is bias. Variance is a super old idea, right? And then, you know, many of you, maybe all of you have heard of bias and variance, but in the era of deep learning, I feel like there have been some changes to the way we think about bias invariance. So I want to talk about some of these ideas which maybe aren't even deep learning per se, but have been slowly shifting as, as we apply deep learning to more and more of our applications. Okay. Oh, and instead of holding all your questions until the end, you know, if you have a question in the middle, feel free to raise your hand as well. I'm very happy to take questions in the middle since this is a more maybe informal whiteboard talk. Right. Oh, and also, you know, say hi to our home viewers. Hi. So, you know, one question that I still get asked sometimes is, and Andre alluded to this earlier, a lot of the basic ideas of deep learning have been around for decades. So why are they taking off just now? Right. Why is it that deep learning, these neural networks we've all known for maybe decades. Why, why are they working so well now? So I think that the one biggest trend in deep learning, the is, is, is scale. That scale drives deep learning progress. Um, and I think Andre mentioned scale of data and scale of computation. Um, and I'll just draw a picture that illustrates that concept maybe a little bit more. Right? So if I plot a figure where on the horizontal axis I plot, um, the amount of data we have for a problem, and on the vertical axis we plot, you know, performance, right? So X axis is the amount of spam data you've collected, Y axis is how accurately can you classify spam? Um, then if you apply, you know, traditional learning algorithms, right, what we found was that the performance often looks like it starts to plateau after a while. Um, it was as if the older generations of learning algorithms, including, you know, support vector logistic regression, SVMs, um, was as if they didn't know what to do with all the data that we finally had. And what happened kind of over the last 20 years, last, last 10 years, was with the rise of the Internet, rise of mobile, rise of IoT, whereas the society sort of marched to the right of this curve, right? For, for, for, for many problems, not all problems. And so with all the buzz and all the hype about deep learning, in my opinion, the number one reason, um, that deep learning algorithms work so well is that if you train, let me call a small neural net, maybe you get slightly better performance. Um, if you train a medium sized neural net, Right, maybe get even better performance. And it's only if you train a large neural net that you could train a model with the capacity to absorb all this data that we all have access to that allows you to get the best possible performance. And so I feel like this is a trend that we're seeing in many verticals, in many application areas. Um, couple comments. One is that, um, this, uh, you know, actually when I draw this picture, some people ask me, well, does this mean a small neural net always dominates a traditional learning algorithm? And the answer is not really. Uh, technically, if you look at the small data regime, if you look at the left end of this plot, right, um, the relative ordering of these algorithms is not that well defined. It depends on who's more motivated to engineer the features better, right? If, if you know, the SVM guy is more motivated to spend more time engineering features, they might beat out the, the, the, the neural network application. But because when you don't have much data, a lot of the knowledge of the algorithm comes from hand engineering. Right, but, so, but this trend is much more evident in the regime of big data where you just can't hand engineer enough features. Uh, and, and, and the large neural net combined with a lot of data tends to outperform. So. Couple of comments. Um, the implication of this figure is that in order to get the best performance, in order to hit that target, uh, you need two things, right? One is you need to train a very large neural network or reasonably large neural network, and you need, um, a large amount of data. And so this in turn has caused pressure to train large net nets, right? Build large nets as well as get huge amounts of data. So one of the other interesting trends I've seen is that increasingly I'm finding that it makes sense to build an AI team as well as build a computer systems team and have the two teams kind of sit next to each other. And the reason I say that is, I guess so. Let's see. So when we started, you know, Baidu research, we set our team that way. Other teams are also organized as well. I think Peter mentioned to me that OpenAI also has a systems team and a machine learning team. And the reason we're starting to organize our teams that way, I think, is that some of the computer systems work we do. Right. So we have an HPC team, high performance team, supercomputing team at Baidu. Some of the extremely specialized knowledge in HPC is just incredibly difficult for, for an AI researcher to learn, right? Some people are super smart, maybe, maybe Jeff Dean is smart enough to learn everything, but, but it's just difficult for any one human to be sufficiently expert in HPC and sufficiently expert in machine learning. And so we've been finding, and Shubo, actually one of the co organizers, is on our HPC team. We've been finding that bringing talent from that knowledge, from these multiple sources, multiple communities, allows us to get our best performance. Um, you know, you've heard a lot of, present a lot of fantastic presentations today. I want to draw one other picture which is, um, in my mind this is how I mentally bucket, you know, work in, in, in deep learning. So this might be a useful categorization, right? When you look at the talk, you can mentally put each talk into one of these buckets I'm about to draw. Um, but I feel like there's a lot of work on, I'm gonna call, you know, General DL or general models. And this would be basically what the type of model that Hugo Larochelle talked about this morning where you have, you know, really densely connected layers. Right. Um, I guess fc, right was, was the. So there's a huge bucket of models there. Um, and then I think a second bucket is sequence models. So 1D sequences. Um, and this is where I would bucket a lot of the work on RNNs, uh, you know, LSTMs, right. Grus, some of the attention models which I guess probably Yoshi Benjamin will talk about tomorrow or maybe, maybe, maybe others, maybe quark as well, I'm not sure. Right. Um, but so the 1D sequence models is another huge bucket. Um, the third bucket is the image models. Um, this is really 2D and maybe sometimes 3D. But this is where I would tend to bucket all the work of CNNs con convolutional nets. And then in my mental bucket then, then there's a fourth one which is the other. Right. And this includes uh, unsupervised learning, you know, the reinforcement learning as well as lots of other creative ideas being explored in literature. And you know, like what I still find slow feature analysis, fast coding, you know, ica, various models kind of in the other category, super exciting. So it turns out that if you look across industry today, almost all the value today is driven by these three buckets. Right? So what I mean is those three buckets of algorithms are you know, driving, causing us to have much better products. Right. Or, or monetizing very well. It's just incredibly useful for lots of things. Um, in some ways I think this bucket might be the future of AI, right. So I, I find unsupervised learning especially super exciting. Uh, so, so I'm, I'm, I, I'm actually super excited about this as well. Um, although I think that if you know, on Monday you have a job and you're trying to like build a product or whatever, the chance of you using something from one of these feedback lists will be, will be highest. Um, but I definitely encourage you to contribute to research here as well. Right. So um, I said that trend one, the, the major trends one of deep learning is scale. Um, this is what I would say is maybe major trend two of, of two of two trends. This is not gonna go on forever, right? Um is, I feel that major trend two is um, the rise of end to end deep learning, uh, for rich es, especially for rich outputs. And so um, end to end deep learning. I'll say a little bit more in a second exactly what I mean by that. But the examples I'm gonna Talk about are all from one of these three buckets, right? General DL sequence models, image 2D, 3D models. Um, but let's see, it's best illustrated a few examples. Um, until recently a lot of machine learning used to output just real numbers, you know. So I guess in Richard's example you have a movie review, right? And then actually but I prepared totally different examples. I was editing my examples earlier to, to, to be more coherent with the speakers before me. Um, but you have a movie review and then output the sentiment. You know, is this a positive or a negative movie review? Um, or you might have an image, right? And then you want to do ImageNet object recognition, you know. So this would be a 01 output. This might be a integer from 1 to 1000. But so until recently a lot of machine learning was about outputting a single number, maybe a real number, maybe an integer. Um, and I think the, the, the number two major trend that we're really excited about is um, end to end deep learning algorithms, they can output much more complex things than numbers. And so one example that you've seen is image captioning where instead of taking an image and saying this is a cat, you can now take an image and output, you know, an entire string of text using RNN to generate that sequence. So I guess what Andre who spoke just now, I think Orio Vendal's Xu Wei at Baidu, right. A whole bunch of people have worked on this problem. Um, one of the things that I guess my, my, my collaborator Adam Coates will talk about tomorrow, maybe Quark as well, not sure is um, speech recognition where you take as input audio and you directly output, you know, the text transcript. Right. And so um, when we first proposed using this kind of end to end architecture to do speech recognition, this is very controversial. We're building my work of Alan's. Great. Uh, but the idea of actually putting this in the production speech system was very, very controversial when we first, you know, said we wanted to do this. But I think the whole community's coming round to this point of view more recently um, or you know, machine translation say go from English to French, right. So the Kwak Ler, others working on this, a lot of teams now, um, or you know, given the parameters, um, synthesize a brand new image. Right. And, and, and you saw some examples of image synthesis. So I feel like the, the, the second major trend of, of, of deep learning that, that I find very exciting. And, and I mean this allowing us to build, you know, transformative things that we just Couldn't build three or four years ago. Has in this trend to what not just learning algorithms can output not just a number that can output very complicated things, things like a sentence or caption or French sentence or image or like the recent WaveNet paper output audio, right? So I think this is the second major trend. So despite all the excitement about end to end deep learning, I think that end to end deep learning, you know, sadly is not the solution to everything. Um, I wanna give you some rules of thumb for deciding when to use what is exactly end to end deep learning and when to use it and when not to use it. So we'll start moving to the second bullet and we'll, we'll, we'll go through these. So the trend towards end to end deep learning has been um, this idea that instead of engineering a lot of intermediate representations, maybe you can go directly from, from your raw input to whatever you wanna predict, right? So for example, actually to take this, I'm, I'm gonna use speech as a recurring example. Uh, so for speech recognition, um, previously one used to go from the audio to you know, hand engineered features like MFCCs or something and then maybe extract phonemes, right? Um, and then eventually you try to generate the transcript. Um, for those of you that aren't sure what a phoneme is. So if you look at the word, listen to the word cat and the word kick, the K sound, right? Is the same sound. And so phonemes are these basic units of sounds, such as K is a phoneme and is hypothesized by linguists to be the basic units of sound. So K a T would be the, maybe the three phonemes that make up the word cat, right? So traditional speech systems used to, used to do this. Uh, and I think 2011, Li Deng and Geoff Hinton, um, made a lot of progress in speech recognition by saying we can use deep learning to do this first step. Um, but the end to end approach to this would be to say let's forget about phonemes, let's just have a neural net, right, Input the audio and output the transcript. Um, so it turns out that in some problems this end to end approach, so one end is the input, the other end is the output. So the phrase end to end deep learning refers to just having a neural net or you know like a learning algorithm directly go from input to output. That's, that's what end to end means. Um, this end to end formula is, I think it makes for what great PR and, and it's actually very simple but it only Works sometimes. Um, and actually maybe this interesting story, you know, this end to end story, we really upset a lot of people. Um, when we were doing this work. I, I guess I used to go around and say, I think phonemes are a fantasy of linguists, um, and we should do away with them. And I still remember there was a meeting at Stanford and some of you know who it was, there was a linguist kind of yelling at me in public for saying that. So maybe, maybe I should not. Uh huh. We turned out to be right. Yeah. All right, um, so let's see. Um, but the, the, the Achilles heel of a lot of deep learning is that you need tons of label data, right? So if, if this is your X and that's your Y, then for end to end deep learning to work you need a ton of label, you know, input output data, X comma, y. So to take an example where, um, where you know, one may or may not consider end to end deep learning, um, this is a problem I learned about just last week from Curtis Langless and, and Darwin, who's in the audience, I think of uh, imagine you want to use um, X ray pictures of your hand in order to predict the child's age, right? So this is a real thing, you know, doctors actually care to look at an X ray of your, of a child's hand in order to predict the age of the child. So um, boy, let me draw an X ray image, right? So this is, you know, the child's hand. So these are the bones, right? I guess this one. Not a doctor. Okay, so that's a hand and, and, and you see the bones. Um, and so more traditional algorithm might input an image and then first, you know, extract the bones. So first figure out, oh, there's a bone here, there's a bone here, there's a bone here and then maybe measure the length of these bones, right? Um, so really I'm gonna say bone length and then maybe has some formula like some linear regression average, some simple thing to go from the bone length to estimate the age of the child, right? So this is a non end to end approach to trying to solve this problem. Um, an end to end approach would be to take an image and then you know, run a convnet or whatever and just try to output the age of the child. And I think this is one example of a problem where it's very challenging to get end to end deep learning to work because you just don't have enough data. You just don't have enough X rays of children's hands. Annotated with their ages. And instead where we see deep learning coming in is in this step, right? To use, go from image to, to figure out where the bones are. Use deep learning for that. But the advantage of this non end to end architecture is, is it allows you to hand engineer in more information about the system such as how bone lengths map to age. Right. Which, which you can kind of get tables about. Um, there are a lot of examples like this and I think one of the unfortunate things about deep learning is that um, let's see. Uh, you know, you can for, for, for suitably sexy va values of x and y, you could almost always train a model and publish a paper. But that doesn't always mean that you know, is actually a good idea. Peter. Maybe it's definition of end to end, but you could in principle people are in the front part knowing what's coming later. I see, yeah, I see, yeah, I see. Yes, that's true. Yes. Peter's pointing out that in practice you could, if this is a fixed function F. Right. You could back prop all the way from the age all the way back to the image. Yeah, that's a good idea actually. Um, who was it just you said you better do it quickly. Uh, yeah, um, let me give a couple other examples. Uh, that, where, where it might be harder to back prop all the way through. Right. So here's, here's an example. Um, take self driving cars. You know, most teams are using an architecture where you input an image, you know, what's in front of the car, let's say, and then you, you know, detect other cars. Right. Uh, and then, and, and maybe use the image, detect pedestrians. Right. Self driving cars are obviously more complex than this. Right. Uh, but then now that you know where the other cars and where the pedestrians are relative to your car, you then have a planning algorithm, uh, to then you know, come up with a trajectory. Right. And then now that you know, um, what's the trajectory that you want your car to drive through? Um, you could then, you know, compute the steering direction. Right. Let's say. And so um, this is actually the architecture that most self driving car teams are using. Um, and you know, there have been interesting approaches to, to say, well I'm gonna input an image and I'll put a steering direction. Right. And I think this is an example of where um, at least with today's data technology, I'd be very cautious about the second approach because, and I think if you have enough data, the second approach will work and you can even prove a theorem you know, showing that it will work, I think. But um, I don't know that anyone today has enough data to make the second approach really, really work well. Right. And, and I think kind of to Peter made a great comment just now and I think, you know, some of these components will be incredibly complicated. You know, like this could be a path plan of exact explicit search and you could actually design a really complicated power plan and generative trajectory and your ability to hand code that still has a lot of value. Right. So this is one thing to watch out for. Um, I have seen project teams say I can get X, I can get Y, I'm gonna train deep learning. Um, but unless you actually have the data, you know, some of these things make for great demos if you cherry pick the examples. But, but it can be challenging to get to work at scale. I, I should say for self driving cars, this debate is still open. I'm, I'm cautious about this. I don't think it's this, I don't think this will necessarily fail. I just think the data needed to do this will be, will be really immense. So I, I, I'd be very cautious about and, and right now, but it might work if you have enough data. Um, so you know, one of the themes that comes up in machine learning already, if you're working on a machine learning project, one thing that will often come up is you will, you know, develop a learning system, train it, maybe it doesn't work as well as you're hoping yet. And the question is, what do you do next? Right. This is a very common part of a machine learning, you know, researcher or machine learning engineer's life, which is, you know, you, you train a model doesn't do what you want it to yet. So what do you do next? Right. This happens to us all the time. Um, and you face a lot of choices. You could collect more data, maybe you wanna train it longer, maybe you want a different neural network architecture, maybe you wanna try regularization, maybe a, you know, bigger model for some more GPUs. So you have a lot of decisions and I think that a lot of the skill of a machine learning researcher, machine learning engineer is knowing how to make these decisions. Right. And, and, and the difference in performance and whether you, you know, do you train a bigger model or do you try regularization, your skill at picking these decisions will have a huge impact on how rapidly you can make progress on natural machine learning problem. So, um, I wanna talk a bit about bias and variance since that's one of the most basic you know, concepts in machine learning. And I feel like it's evolving slightly in the era of, of, of deep learning. So to use a, as a motivating example, um, let's say the goal is to build a human level. Right. Speech system. Right. Speech recognition system. Okay? So, um, what we would typically do, especially in academia, is we'll get a data set. You know, here's my data set with a lot of examples. And then we shuffle it and we randomly split it into 70, 30 train tests, or maybe, or maybe 70% train, you know, 15% dev and 15% test. Right, right. Oh, and some people use the term validation set, but I'm, I'm just gonna use the set. The dev set or stands for development set means the same thing as validation set. Okay? So it's pretty common. Um, and so what we would. What, what, what I encourage you to do, if you aren't already, is to measure the following things. Um, human level error. So let, let, actually let me illustrate an example. Let's say that on your devel. Uh, uh, let's say that, um, on your dev set, you know, human level error is, uh, 1% error. Um, let's say that your training set error is, um, is 1% error. Um, let's say that your training set error is. Let me use 5%.