deeplearning.ai's Heroes of Deep Learning: Yoshua Bengio

Hi, Joshua. I'm really glad you could join us here today.

Oh, I'm very glad too.

You know, today you're not just a researcher or engineer in deep learning. You've become one of the institutions or one of the icons of deep learning. But I'd really like to hear the story of how it started. So how did you end up, you know, getting into deep learning and then pursuing this journey?

Right. Well, actually it started when I was a kid, adolescent, reading a lot of science fiction, like I guess many of us. And when I started my graduate studies in 1985, I started reading Neolette papers and that's where I got all excited and it became really a passion.

And actually, what was that like in what, mid-80s, right, 1985, reading these papers, do you remember?

Yeah, it was, well, you know, coming from the courses I had taken in classical AI with expert systems and suddenly discovering that there was all this world of thinking about how humans might be learning and human intelligence and how we might draw connections between that and artificial intelligence and computers. That was really exciting for me when I discovered this literature and I started reading the connectionists, of course, so the papers from Geoff Hinton, Ruma Hart and so on. I worked on recurrent nets, I worked on speech recognition, I worked on, hmm, so graphical models. And then quickly I moved to AT&T, Bell Labs and MIT where I did postdocs. And that's where I discovered some of the issues with long term dependencies with training neural nets. And then shortly after I got recruited UDM back in Montreal, where I had spent most of my adolescence years.

So as someone who's been there for the last several decades and seen it all, or certainly seen a lot of it, um, tell me a bit about how you're thinking about deep learning about neural networks as evolved over this, over this time.

We start with experiments with intuitions and theory sort of comes later and we now understand a lot better, for example, why backprop is working so well, why depth is so important. And, you know, these kinds of notions we didn't have any solid justification for in those days. When we started working on Deep Net in the early 2000s, we had the intuition that it made a lot of sense that a deeper network should be more powerful, but we didn't know how to take that and prove it. And of course our experiments initially didn't work.

And actually what were the most important things that you think turned out to be right and what were the biggest surprises of what turned out to be wrong compared to what we knew 30 years ago.

Oh, sure. So one of the biggest mistake I made was to think that, like everyone else in the 90s, that you needed smooth nonlinearities in order to. For backprop to work. Because I thought that, you know, if we had something like the rectifying nonlinearities, where you have a flat part, that it would be really hard to train because the derivative would be zero in so many places. And when we started experimenting with RELU with deep nets in around 2010, I was obsessed with the idea that, oh, we should be careful about whether neurons won't saturate too much on the zero part. But in the end, it turned out that actually the RELU was working a lot better than the sigmoids and attach. And that was a big surprise. We did this exploring this because of the biological connection, actually, not because we thought that it would be easier to optimize, but it turned out worked better, whereas I thought it would be harder to try.

So let me ask you, what is the relationship between deep learning and the brain? There's the obvious answer, but I'm curious, what's your answer to that?

Well, the initial insight that really got me excited with neural nets was this idea from the connectionists that information is distributed across the activation of many neurons, rather than being represented by sort of the grandmother cell, as they were calling it, where a symbolic representation. That was the traditional view in classical AI. And I still believe this is a really important thing. And I see people discovering the importance of that even recently. So that was really a foundation. The depth thing is something that came later in the early 2000s, but it wasn't something I was thinking about in the 90s, for example.

Right. I remember you built a lot of relatively shallow but very distributed representations for the word embeddings very early on.

That's right, yeah. That's one of the things that I got really excited about in the late 90s with. Actually, my brother Sandy and I worked on the idea that we could use neural nets to tackle the curse of dimensionality, which was believed to be one of the central issues with statistical learning. And the fact that we could have these representations could be used to represent joint distributions over many random variables in a very efficient way. And it turned out to work quite well. And then I extended this to join distributions over sequences of words. And this is how the word embeddings were born, because I thought, oh, this will allow generalization across words that have similar semantic meaning and so on.

So over the last couple decades, your research group has invented more ideas than Anyone can summarize in a few minutes. So I'm curious, what are the inventions or ideas you are most proud of from your group?

Right. So I think I mentioned long term dependencies, the study of that. I think people still don't understand it well enough. Then there's the story I mentioned about curse of dimensionality joint distributions with neural nets, which became more recently the nade that Hugo Dan Rochelle did. And then as I said, that gave rise to also the work on learning word embeddings for joint distributions for words. Then came I think probably the best known events of the work we did with deep learning with stacks of autoencoders and stacks of RBMs. One thing then was the work on understanding better the difficulties of training deep nets with Xavi Laurel with the initialization ideas and also the vanishing gradient in deep nets. And that work actually was the one which gave rise to the experiments showing the importance of piecewise linear activation functions. Then I would say some of the most important work regards the work we did with unsupervised learning. The denoising autoncoders, the GANs, which are very popular these days, the narrative adversarial networks, the work we did with neural machine translation using attention, which turned out to be really important for making attention, making translation work and is currently used in industrial systems like Google Translate. But this attention thing actually really changed my views on neural nets. Neural nets, we used to think as machines that can map a vector to a vector, but really with attention mechanisms you can now handle any kind of data structure. This is really opening up a lot of interesting avenues, direction of actually connecting to biology. One thing that I've been working on in the last couple of years is how could we come up with something like backprop, but that brains could implement. And we have a few papers in that direction that seems to be interesting for the neuroscience people and then we're continuing on that direction. Of course.

One of the topics that I know you've been thinking a lot about is the relationship between deep learning and the brain. Can you tell us a bit more about that?

The biological thing is something I've been thinking about for a while actually and having a lot of, I would say daydreaming about because I think of it like a puzzle. So we have these pieces of evidence from what we know from the brain and from learning in the brain, like spike timing, dependent plasticity. And on the other hand we have all of these concepts from machine learning. The idea of globally training the whole system with respect to an objective function and the idea of backprop. And what does backprop mean? Like, what does credit assignment really mean? When I started thinking about how brains could do something like backprop, it prompted me to think about, well, maybe there's some more general concepts behind backprop which make it so efficient, which allow us to. To be efficient with backprop. Maybe there's a larger family of ways to the credit assignment and that connects to questions that people in reinforcement learning have been asking. So it's interesting how sometimes asking a simple question leads you to thinking about so many different things and forces you to think about so many elements that you'd like to bring together like a big puzzle. So this has gone for a number of years. I need to say that this whole endeavor, like many of the ones that I've followed, has been highly inspired by Geoff Hinton's thoughts. So in particular, he gave this talk in 2007, I think the first deep learning workshop on what he thought was the way that the brain is working, how temporal code could be used for potentially doing some of the job of backprob. And that led to a lot of the ideas that I've explored in recent years with this. Yeah. So it's kind of an interesting story that has been running for a decade now.

Basically, one of the topics I've heard you speak about multiple times as well is unsupervised learning. Can you share your perspective on that?

Yes. Yes. So unsupervised learning is really important right now. Our industrial systems are based on supervised learning, which essentially requires humans to define what the important concepts are for the problem and to label those concepts in the data. And, you know, we build all these amazing toys and services and systems using this, but. But humans are able to do much more. They're able to explore and discover new concepts by observation and interaction with the world. A two year old is able to understand intuitive physics. In other words, she understands gravity, she understands pressure, she understands inertia, she understands liquid solids. And of course, her parents never told her about any of these stuff. Right. So how did she figure it out? So that's the kind of question that unsupervised learning is trying to answer. It's not just about we have labels or we don't have labels. It's about actually building a construction, a mental construction that explains how the world works by observation. And more recently, I've been combining the ideas in unsupervised learning with the ideas in reinforcement learning, because I believe that there's a very strong Indication about the important underlying concepts that we're trying to disentangle. We're trying to separate from each other that a human or machine can get by interacting with the world, by exploring the world and trying things and trying to control things. So these are, I think, tightly coupled to the original ideas of unsupervised learning. So my take on unsupervised learning 15 years ago, when we started doing the auto encoders and the RBMs and so on, was very focused on the idea of learning good representations. And I still think this is a central question, but the thing we don't know is how. What is a good representation? How do we figure out an objective function, for example? So we've tried many things over the years, and that's actually one of the cool things about unsupervised learning research, that there are so many different ideas, so different ways that this problem can be attacked, and that suggests maybe there's another one we'll discover, you know, next year that's completely different, and maybe the brain is using something else completely different. So it's not incremental research. It's something that in itself is very exploratory. We don't have, like, a good definition of what's the right objective function to even measure that a system is doing a good job of unsupervised learning. So, of course it's challenging, but at the same time, there's, you know, it leaves open a wide field of possibilities, which is what, you know, researchers really love. At least that's something that appeals to me.

So, you know, today there's so much going on in deep learning, and I think we've passed the point where it's possible for any one human to read every single deep learning paper being published. Um, so I'm curious, what in deep learning today excites you the most?

So I'm very ambitious, and I feel like the current state of the science of deep learning is far from where I'd like to see it. And I have the impression that our systems right now make mistakes that suggest the kind of mistakes that suggest that they have a very superficial understanding of the world. So what excites me the most now is sort of direction of research where we're not trying to build systems that are going to do something useful. We're just going back to principles about how can a computer observe the world, interact with the world, and discover how that world works, even if that world is simple, something that we can program as a kind of video game. We don't know how to do that well, and that's cool because I don't have to compete with Google and Facebook and Baidu and so on. Right? Because this is a kind of basic research that can be done by anyone in their garage and could change the world. So there are many, of course, many directions to attack this. But I see a lot of the fruitful interactions between ideas and deep learning and reinforcement learning being really important there. And I'm really excited that the progress in this direction could have a huge impact on practical applications, actually, because if you look at some of the big challenges that we have in applications, like how we deal with new domains or categories on which we have too few examples, and in cases where humans are very good at solving those problems. So these transfer learning and generalization issues, they would become much easier to tackle if we had systems that had a better understanding of how the world works, a deeper understanding what is actually going on, what are the causes of what I'm seeing and how could I influence what I'm seeing by my actions? So these are the kinds of questions I'm really excited about these days. And I think they connect also the deep learning research that has evolved over the last couple of decades with even older questions in AI because a lot of the success in deep learning has been with perception. So what's left? Right? What's left is sort of higher level condition, which is about understanding at an abstract level how things work. So we, our program of understanding high level abstractions, I think, has, you know, not reached those high level of abstractions. And so we have to get there. We have to think about reasoning about sequential processing of information. We have to think of how causality works and, and how machines can discover all these things by themselves, potentially guided by humans, but as much as possible in an autonomous way.

And it sounds like from part of what you said that you're a fan of research approaches where you experiment on, you know, I'm gonna use the term toy problem, not in a disparaging way, but on the small problem. And you're optimistic that that transfers to bigger problems later?

Yes, yes, yes. I mean, it transfers in a, in a meta way. Right. Of course we're gonna have to do some work to scale up and address those problems. But, but my main motivation for going for those toy problems is that we can understand better our failures and we can reduce the problem to something we can intuitively sort of manipulate and understand more easily. So sort of a classical divide and conquer science approach. And also I think something people think about Enough is the cycle, the research cycle can be much faster. Right. So if I can do an experiment in a few hours, I can progress much faster if I have to train a huge model that tries to capture the whole common sense and everything in the general knowledge, which eventually will do. Each experiment just takes too much time with current hardware. So while our hardware friends are building machines that are going to be a thousand or million times faster, I'm doing those toy experiments.

You know, I've also heard you speak about the science of deep learning, not just as an engineering discipline, but doing more work to understand what's really going on. Do you want to share your thoughts on that?

Yeah, absolutely. I fear that a lot of the work that we're doing is sort of like blind people trying to find their way. And you know, you can get a lot of luck and find interesting things that way. But, but really if we sort of stop a little bit and, and try to understand what we're doing in, in a way that's transferable because, because we go down to principles, to theory. But when I say theory, I don't mean necessarily math, right? I don't, I'm not like, of course I, I like math and so on, but, but I don't think that we need that everything be formalized mathematically, but be formalized logically in other sense that I can convince somebody that, you know, this should work with, this makes sense, this is the most important aspect. And then math allows us to make that stronger and tighter. But really it's more about understanding and it's about also doing our research not to beat the next baseline or a benchmark, or beat the other guys in the other lab or the other company. It's more about what kind of question should we ask that would allow us to understand better the phenomena of interest? What makes, for example, training in deeper networks harder or in recurrent nets harder? We have some ideas, but a lot of things we don't understand yet. So we can maybe design experiments whose goal is not to have like a better algorithm, but just to understand better the algorithms we currently have, or, you know, why. What circumstances make particular algorithm work better and why. I mean, it's the why that really matters. That's what science is about.

It's why right today there are a lot of people that want to enter the field. And I'm sure you've answered this a lot in one on one settings, but you know, with, with all the people watching this on video, what advice would you have for people that want to get into AI get into deep learning.

Right. So, so first of all, there are different motivations and different things you could do. You know, what you need to become a deep learning researcher may not be the same as if you want to be an engineer who's going to use deep learning to build products. There's a different level of understanding that's needed in both cases. But, but in any case, in both cases, practice, practice. So to really master a subject like deep learning, of course you have to read a lot, you have to practice so programming the things yourself. Very often I, you know, I interview students who have used software and these days there's so, so many, you know, so, so good softw. You can just plug and play and understand nothing of what you're doing or at such a superficial level that then it becomes hard to figure out when it doesn't work, what's going wrong. So actually trying to implement things yourself, even if it's inefficient, but just to make sure you really understand what is going on is really useful and trying yourself.

So don't just use one of the programming frameworks. You can do everything in a few lines of code, but you don't really know what just happened.

Right, exactly, exactly. And I would say even more than that, try to derive the thing yourself from first principles if you can. So that really helps. But I mean the usual things you have to do, like reading, looking at other people's code, writing your own code, doing a lot of experiment, making sure you understand everything you do. So I mean, especially for the science part of it, trying to ask why am I doing this? Why are people doing this? And maybe the answer is somewhere in the book and you have to read more, but it's even better if you can actually figure it out by yourself.

And in fact, of the things I read, you and Ian Goodfellow and Aaron Colville wrote a very highly regarded book.

Thank you. Thank you. Yes, it's selling a lot. It's a bit crazy. I feel like there's more people reading this book than people who can read it right now. But yeah, also proceedings of the ICLR iclear conference is probably the best concentrated place of good papers. Of course there are really good papers at NIPS and ICML and other conferences. But if you really want to go for a lot of good papers, just read the last few ICLR proceedings and that will give you a really good view of the field.

Cool.

Yeah.

Any other thoughts? When people ask you for advice? You know, how does someone.

Right.

Become good at deep learning?

Well, it depends on where you come from. Don't be afraid by the math. Just, you know, just develop the intuitions and then the math become really easier to understand once you, once you get the hang of what's going on at an intuitive level. And one good news is that you don't need 5 years of PhD to become proficient at deep learning. You can actually learn pretty quickly. If you have a good background in computer science and math, you can learn enough to use it and build things and start research experiments in just a few months, something like six months. For people with the right training, maybe they don't know anything about machine learning, but if they're good in math and computer science, it can be very fast. And of course, so that means you need to have the right training in math and computer science. Sometimes what you learn in just computer science courses is not enough. You need some continuous path especially so this is probability, algebra and optimization for example.

And calculus.

And calculus, yeah.

Thanks a lot Joshua, for sharing all the comments and insights and advice. Even though I've learned, even though I've known you for a long time, there are many details of your early history that I didn't know until now. So thank you.

Well, thank you Andrew, for doing this special recording and what you're doing and well, I hope it's going to be used by a lot of people.