deeplearning.ai's Heroes of Deep Learning: Yann LeCun

Hi, Jan. You've been such a leader for deep learning for so long. Thanks a lot for doing this with us.

Well, thanks for having me.

So you've been working on neural nets for a long time. Um, I'd love to hear your personal story of how did you get started in AI? How did you networking with neural networks?

So I got interested in. I was always interested in the intelligence, as in general, like the emergence of intelligence in. In humans, you know, that got me interested into, you know, human evolution when I was a kid. And u.

This is in France.

I was in France. I was in middle school or something and I was interested in technology and space, etc. My favorite movie was 2001 A Space Odyssey. You had intelligent machines, space travel and human evolution as kind of the themes. That was what I was fascinated by. The concept of intelligent machine, I think really appealed to me. And then I studied electrical engineering. And when I was at school, I was maybe in second year of engineering school, I stumbled on a book which was actually a philosophy book. It was a debate between Noam Chomsky, the computational linguist at mit, and Jean Piaget, who is cognitive psychologist, sort of psychology of child development in Switzerland. And it was basically a debate between nature and nurture where Chomsky arguing for the fact that language has a lot of innate structure and Piaget saying a lot of it is learned, et cetera. And on the side of Piaget was a transcription of a person. Each of these guys sort of brought their teams of people to argue for their side. And on the side of Piaget was Seymour Peppert from MIT who had worked on the Perceptron model, one of the first machines capable of running. And I never heard of the Perceptron. And I read this article and say, a machine capable of running, that sounds wonderful. And so I started going to several university libraries and searching for everything I could find that talked about the Perceptron and realized there was a lot of papers from the 50s, but it kind of stopped at the end of the 60s with a book that was co authored by the same Seymour Papert.

What year was this then?

So this was 1980, roughly. And so I specialized. I did a couple projects with some of the math professor in my school on kind of neural nets, essentially. But there was no one I could talk to who had worked on this because the field basically had disappeared in the meantime. Right. 1980, nobody was working on this and experimented with this a little bit, writing simulation software, various kinds of reading about neuroscience. When I finished my engineering studies I studied chip design, UVSI design at the time, so it's something completely different. And when I finished, I really wanted to kind of do research on this. And I figured out already that at the time that the important question was how you train neural nets with multiple layers. It was pretty clear in the literature of the 60s that that was the important question that had been left unsolved and the idea of hierarchy and everything. You know, I'd read Fukushima's article on the neocognitron, right, which was this sort of hierarchical architecture, very similar to what we now call commercial nets, but, you know, without the backprop style learning algorithms. And I met a small. I met people who were in a small independent lab in France that they were interested in what they called at the time automata networks. And they gave me a couple papers, the people on hubfield networks, which is not very popular anymore, but it's the first associative memories with neural nets. And that paper kind of revived the interest of some research communities into neural nets in the early 80s by mostly physicists and condensed matter physicists and a few psychologists. It was still not okay for engineers and computer scientists to talk about neural nets. And they also showed me another paper that had just been distributed as a preprint, whose title was Optimal Perceptual Inference. And this was the first paper on Boltzmann machines by Jeff Hinton and Teresinowski. And he was talking about hidden units. He was talking about basically the problem of learning, you know, multi layer neural nets that are more capable than just linear classifiers. So I said, I need to meet these people because they are really interested in the right problem. And a couple years later, after I started my PhD, I participated in a workshop in Les Hushes and that was organized by the people I was working with. And Terry was one of the, one of the speakers at that workshop. So I met him at that time.

This is like early 80s now.

This is 1985, early 1985. So I met Teriusinowski in 1985 in the workshop in France in Les Hushes. And a lot of people were there from the sort of early neural net jump up field and a lot of people working on theoretical neuroscience and stuff like that. It was a fascinating workshop. I met also a couple of people from Bell Labs who eventually hired me at Bell Labs, but this was several years before I finished my PhD. So I talked to Terry Sinofsky and I was telling him about what I was working on, which was some version of backprop at the Time. This was before backprop was a paper. And Terry was working on nettalk at the time. This was before the Rumalhart Hinton Williams paper on backprop had been published. But he was friends with Jeff, this information was circulating. So he was already working on trying to make this work for NetTalk. But he didn't tell me. And he went back to the US and told Jeff, there is some kid in France who is working on the same stuff we were working on.

I see.

And then a few months later, in June, there was another conference in France where Jeff was a keynote speaker and he gave a talk on Bolson machines. Of course, he was working on the backboard paper. And he gives this talk and then there is 50 people around him who want to talk to him. And the first thing he says to the organizer is, do you know this guy Yann Lecun? And it's because he had read my paper in the proceedings that was written in French and he could sort of read French and he could see the math and he could figure out it was sort of backprop. And so we had lunch together and that's how we became friends.

I see. Wow. So that's because you basically, you know, independently, multiple groups independently reinvented or invented backprop pretty much.

Right. Or realize that the whole idea of chain rule, or what the optimal control people call the adjoint state method, which is really the context in which backprop was really invented, this is the context of optimal control back in the early 60s. This idea that you could use gradient descent basically with multiple stages, is what backprop really is. And that popped up in various contexts at various times and. But I think, you know, the Werner Hart Hinton Williams paper is the one that popularized it. I see. Yeah.

Cool. And then, you know, fast forward a few years, you wound up at AT&T Bell Labs, where you invented, among many things, the Net, which we talk about in the. In the.

In the course.

And I remember when, way back, I was a summer intern at AT&T Bell Labs, work of Michael Kearns and a few others, and hearing about your work even back then, so. So tell me more about your AT&T learnet experience.

So what happened is I actually started working on CommercialNet when I was a postdoc at University of Toronto with Jeff Hinton. I did the first experiment, I wrote the code there and did the first experiments there that showed that if you had a very small data set, the data set I was training on, there was no MNIST or anything. Like that back then. So I drew a bunch of characters with my mouse. I had an Amiga personal computer, which was computer ever, and I drew a bunch of characters and then used that augmentation to kind of increase it, and then use that as a way to test the performance. And I compared things like fully connected nets, locally connected nets without shared weights, and then shared weight networks, which is basically the first convnet. And that worked really well for relatively small data sets, could show that you get better performance and no overtraining with conventional architecture. And when I got to Bell Labs in October 1988, the first thing I did was first scale up the network because we had faster computers. A few months before I got to Bell Labs, my boss at the time, Larry Jackel, who became my department head of Bell Labs, said, oh, we should order a computer for you before you come. What do you want? I said, well, you know, here in Toronto There is a Sun 4, which was the latest, greatest stuff. It'd be great if we had one. And they ordered one and I had one for myself at University of Toronto. It was one for the entire department, one just for me. And so what Larry told me, he said, yeah, you know, about you don't get famous by saving money. So that was awesome. And they had been working already for a while on character recognition. They had this enormous Data set called USPS that had 5,000 training samples. And so immediately I trained, designed a convolutional net, which was Learnet one, basically, and trained it on this data set and got really good results, better results than the other methods they had tried on it and that other people had tried on this dataset. So we knew we had something fairly early on. This was within three months of me joining, joining Bell Labs. And so that was the first version of convolutional Net where we had a convolution with stride and we did not have separate subsampling and pooling layers. So each convolution was actually subsampling directly. And the reason for this is that we just could not afford to have a convolution at every location. There was just too much computation. So the second version had a separate convolution and pooling layer and subsampling. I guess that's the one that's called Lunet 1, really. We published a couple of papers on this in neural computation and at NIPs. And so one interesting story. I gave a talk at NIPS about this work and Geoff Hinton was in the audience. And then I came back to my seat. I was sitting next to him and he said, you know, there's one bit of information in your, in your talk which is that if you do all the sensible things, it actually works.

And then in fact, shortly after, a dead line of work went on to make history because it became widely adopted. These ideas became widely adopted for reading checks.

Yeah. So they became widely adopted within AT&T, but not very much outside. And you know, I think it's a little difficult for me to really understand why, but there's several factors, I think. So, you know, this was back in the late 80s and there was no Internet. You know, we had email, we had FTP, but there was no Internet really. Nobody, you know, no two labs were using the same software or hardware platform. Right. You know, some people had some workstations, others had, you know, other machines. You know, some people were using PCs, whatever. There was no such thing as Python or MATLAB or anything like that. People were writing their own code. I had spent a year and a half basically writing me and Leon Botoux, when he was still a student, we were working together and we spent a year and a half basically just writing a neural net simulator. And at the time, because there was no matlab or Python, you had to write your own interpreter to control it. So we wrote our own Lisp interpreter. And so all the learned was written in Lisp using a numerical backend, very similar to what we have now with blocks that you can interconnect and automatic differentiation, all that stuff that we familiar with now with Torch and PyTorch and TensorFlow and all those things. So then we developed a bunch of applications. We got together with a group of engineers, very smart people. Some of them were like theoretical physicists who kind of turned engineer at Bell Labs. Chris Burgess was one of them, who then had a distinguished career at Microsoft Research afterwards and Craig Noll and a bunch of other people. And we were collaborating with them to kind of make this technology practical. And so they developed together, we developed those character recognition systems. And that meant integrating convolutional nets with things like, similar to things that we now call CRFs, for interpreting sequences of characters, not just individual characters.

Yeah, right. The Net paper had partially on the neural network and partially on the autonomous machinery to put it together.

Yeah, that's right. And so the first half of the paper is on convolutional nets and the paper is mostly cited with for that. And then the second half, very few people have read it and it's about sort of sequence level discriminative learning. And basically structured prediction without normalization. So it's very similar to crf. In fact, WITH preceded CRF by several years. So that was very successful, except that the day we were celebrating the deployment of that system in a major bank, we worked with this group that was mentioning that was kind of doing the engineering of the whole system. And then another product group in a different part of the country that belonged to a subsidiary of AT&T called NCR. So this is the national cash Register. National cash register, right. They also build large. I mean, they build ATM machines and they build large check reading machines for banks. So they were the customers, if you want. They were using our check reading systems and they deployed it in a bank. I can't remember which bank it was. They deployed also the ATM machines in the French bank so they could read the check you would deposit. And we were all at a fancy restaurant celebrating the deployment of this thing when the company announced that it was breaking itself up. So this was 1995, and ATT announced that it was breaking itself up into three companies. So there was AT&T and then there was Lucid Technologies and NCR. So NCR was spun off and Lucent Technologies was spun off. And the engineering group went with Lucent Technologies and the product group, of course, went with NCR. And the sad thing is that the AT&T lawyers, in their infinite wisdom assigned the patents of. There was a patent on convolutional nets.

I see.

Which is thankfully expired.

I see.

Expired in 2007 and about 10 years ago. And they signed a patent to NCR. But there was nobody in NCR who actually knew even what a corporation was really. I see. And so the patent was in the hands of people who had no idea what they had. And we were in a different company that now could not really develop the technology. And then our engineering team was in a separate company because we went with at&t. The engineering went with Lucent and the product group with ncr. So it was a little depressing, I see.

Yeah. So in addition to your early work when neural networks were potential, you kept persisting on neural networks even when there was some sort of winter for neural nets. What was that like?

Well, so I persisted and didn't persist. In some ways I was always convinced that eventually those techniques would come back to the fore and sort of, you know, people would figure out how to use them in practice and it would be useful. So I always had that in the back of my mind. But in 1996, when AT&T broke itself up and all of our work on character recognition basically was kind of broken up because the product group went a separate way. I was also promoted to a department head and I had to figure out what to work on. And this was the early days of the Internet. We're talking in 1995. And I had the idea somehow that one big problem about the emergence of the Internet was going to be to bring all the knowledge that we had on paper to the digital world. And so I started actually a project called Dejavu djvu, which was to compress scanned documents essentially so they could be distributed over the Internet. And this project was really fun for a while and had some success, although AT&T really didn't know what to do with it.

Yeah, I remember that it was helping dissemination of online research papers.

Yeah, that's right, exactly. And we scanned the entire proceedings of NIPs and we made them available online to kind of demonstrate how that worked. And we could compress high resolution pages to just a few kilobytes.

So confidence, starting from some of your much earlier work has now come and pretty much taken over the field of computer vision and starting to encroach significantly into even other fields. So just tell me about how you saw that whole process.

So tell you how I thought this was going to happen early on. So, first of all, I always believed that this was going to work. It required fast computers and lots of data. But I always believed somehow that this was going to be the right thing to do. What I thought originally when I was at Bell Labs, that there was going to be some sort of continuous progress along these directions as machines got more powerful. And we were even designing chips to run convolutional nets at Bell Labs. Bernard Boser actually and Hans Peter Graffiti separately had two different chips for running commercial nets really efficiently. And so we thought there was going to be kind of a pickup of this and kind of growing interest and sort of continuous progress for it. But in fact, because of the sort of interest for neural nets that are dying in the mid-90s, that didn't happen. So it was kind of a dark period of seven or eight, six or seven years between 1995 roughly and 2002, when basically nobody was working on this, in fact, there was a little bit of work, there was some work at Microsoft in the early 2000s on using convolutional nets for Chinese character recognition. Patrice Martrice Simard's group. Yeah, exactly. And some other small work for phase detection and things like this in France and in various other places. But it was very small. I discovered actually recently that there's a couple groups that came up with ideas that are essentially very similar to commercial nets, but never quite published it the same way for medical image analysis. And those were mostly in the context of commercial systems. And so it never quite made it to the population. It was after our first work on commercial nets and they were not really aware of it, but. But it sort of developed in parallel a little bit. So several people got kind of similar ideas several years interval. But then I was really surprised by how fast interest picked up after the Imagenet 2012. In 2012. So it's the end of 2012. It was kind of a very interesting event at ECCV in Florence where there was a workshop on Imagenet and everybody knew that Jeff Hinton's team, Alex Krizevsky and Ilya Suskever, had won by a large margin. And so everybody was waiting for Alej Krizewski's talk. And most people in the computer vision committee had no idea what convolutional net was. I mean, they heard me talk about it. I actually had an invited talk at CBPR in 2000 when I talked about it, but most people had not paid much attention to it. Senior people did. They knew what it was, but the more junior people in the community really had no idea what it was. And so Alex Krizevsky just gives his talk and he doesn't explain what a convergent net is because he assumes everybody knows because it comes from machine learning. So he says, here is how everything is connected and how we transform the data and what results we get, assuming that everybody knows what it is. And a lot of people are incredibly surprised. And you could see the opinion of people changing as he was kind of giving his talk. Very senior people in the field.

So you think that workshop was the defining moment that swayed a lot of the computer vision community?

Yeah, definitely. That's what happened right there.

So today you retain a faculty position at NYU and you also lead fair, Facebook and AI research. I know you have a pretty unique point of view on how corporate research should be done. Do I share your thoughts on that?

Yeah. So, I mean, one of the beautiful things that I managed to do at Facebook in the last four years is that I was given a lot of freedom to set up fair the way I thought was the most appropriate because this was the first research organization within Facebook. Facebook is a sort of engineering centric company and so far was really focused on sort of survival or short term things. And Facebook was about to turn 10 years old. Had a successful IPO and was basically thinking about the next 10 years. Right? I mean, Mark Zuckerberg was thinking, what is going to be important for the next 10 years? And the survival of the company was not in question anymore. So this is the kind of transition where a large company can start to think, or it was not such a large company at the time. Facebook had 5,000 employees or so, but it had the luxury to think about the next 10 years and what would be important in technology. And Mark decided that. Mark and his team decided that AI was going to be crucial piece of technology for connecting people, which is the mission of Facebook. And so they explored several ways to kind of build an effort in AI. They had a small internal group, engineering group experimenting with conversion nets and stuff that were getting really good results on face recognition and various other things which piqued their interest. And they explored the idea of hiring a bunch of young researchers or acquiring company or things like this. And they settle on idea of hiring someone senior in the field and then setting up a research organization. And it was a bit of a culture shock initially because the way research operates in the company is very different from engineering. You have longer time scales and horizon. And researchers tend to be very conservative about the choice of places where they want to work. And I made very clear very early on that research needs to be open, that researchers need to not only be encouraged to publish, but be even mandated to publish and also be evaluated on criteria that are similar to what we use to evaluate academic researchers. And so what Mark and Mark Schaefer, the CTO of the company, who is my boss now, said, they said, Facebook is a very open company. We distribute a lot of stuff in open source. Schrep, the cto, comes from the open source world. He was Mozilla before that, and a lot of people came from that world. So that was in the DNA of the company. So that made me very confident that we could set up an open research organization. And then the fact that the company is not obsessively compulsive about IP as some other companies are, makes it much easier to collaborate with universities and have arrangements by which a person can have a foot in industry and a foot in academia.

And you find that valuable yourself?

Oh, absolutely, yeah. So if you look at my publications over the last four years, the vast majority of them are publications with my students at nyu, because at Facebook I did a lot of organizing the lab, hiring, scientific direction and advising and things like this, But I don't get involved in individual research projects to get my name on papers, and I don't care to get my name on papers anymore.

But it's about setting up someone else to do great work rather than doing all the great work yourself.

And you never want to put yourself, you know, you want to kind of stay behind the scene and you don't want to put yourself in competition with people you know in your lab. In that case, I'm sure you get.

Asked this a lot, but hope you can answer for all the people watching this video as well. What advice do you have for someone wanting to get involved in AI or break into AI?

I mean, it's such a different world now than it was when I got started. I think what's great now is it's very easy for people to get involved at some level. The tools that are available are so easy to use now with TensorFlow, PyTorch, whatever, you can have a relatively cheap computer in your bedroom and basically train your convolutional net or your current net to do whatever. And there's a lot of tools you can learn a lot from online material about this without it's not very onerous. So you see high school students now playing with this. Right. Which is great, I think. And there's certainly a growing interest from the student population to learn about machine learning and AI. And it's very exciting for young people. And I find that wonderful, I think. So my advice is, if you want to get into this, make yourself useful. So make a contribution to an open source project, for example, or make an implementation of some standard algorithm that you couldn't find the code of online, but you'd like to make it available to other people. So take a paper that you think is important and then reimplement the algorithm and then put it up in open source package or contribute to one of those open source packages. And if the stuff you write is interesting or useful, you'll get noticed. Maybe you'll get a nice job at a company you really want a job at, or maybe you'll get accepted in your favorite PhD program or things like this. So I think that's a good way to get started.

So open source contributions is a good way to enter the community, learn.

That's right.

Thanks a lot, Jan. That was fascinating. I've known you for many years. It's still fascinating to hear all these details of all the stories I've gone on over the years.

Yeah, there's many stories like this that reflecting back at the moment when they happen, you don't realize what importance they might take 10 or 20 years later. Thank you. Thanks.