deeplearning.ai's Heroes of Deep Learning: Ruslan Salakhutdinov

Welcome, Russ. I'm really glad you could join us here today.

Thank you. Thank you, Andrew.

So, you know, today you're the director of research at Apple and you also have a faculty, a professor at Carnegie Mellon University. So I'd love to hear a bit about your personal story. How did you end up doing this deep learning work that you do?

Yeah, it's actually, to some extent, it was. I started in deploying, to some extent, by luck. I did my master's degree at Toronto and then I took a year off. I was actually working in the financial sector. It's a little bit surprising. And at that time, I wasn't quite sure whether I want to go for my PhD or not. And then something happened. Something surprising happened. I was going to work one morning and I bumped into Geoff Hinton and Jeff told me, hey, I have this terrific idea. Come to my office, I'll show you. And so we basically worked together and he started telling me about, you know, these Boltzmann machines and contrastive divergence and some of the tricks, which I didn't at that time quite understand what he was talking about. But that really, really excited was very exciting and really excited me. And then basically within three months, I started my PhD with Jeff. So that was kind of like the beginning, because that was back in 2005, 2006. And this is where some of the original deep learning algorithms using restricted BOSS machines and supervised per training were kind of popping up. And so that's how I started. It was really that one particular morning when I bumped into Jeff completely changed my future career moving forward.

And then, in fact, you were a co author on one of the very early papers on restricted Boltzmann machines that really helped with this resurgence of neural networks and deep learning. Tell me a bit more what that was like. You're working on that seminal.

Yeah, this was. This was actually a really. This was exciting year. It was a first year. It was my first year as a PhD student. And Jeff and I were trying to explore these ideas of using restricted Boltz machines and using pre training tricks to train multiple layers. And specifically we were trying to focus on autoencoders. You know, how do we do nonlinear extension of PCA effectively? And it was very exciting because we got these systems to work on MNIST digits, which was exciting. But then the next steps for us were to really see whether we can extend these models to dealing with faces. I remember we had this Oliveta faces data set and then we started looking at can we do compression for documents. We started looking at all These different data, real value, count, binary. And throughout a year, I was a first year PhD student, so it was a big learning experience for me. But really within six or seven months we were able to get really interesting results and really good results. Something that we were able to train these very deep autoencoders. This is something that you couldn't do at that time, but using sort of traditional optimization techniques. And then it turned out into really, really exciting payoff for us. That was super exciting year because it was a lot of learning for me. But at the same time, the results turned out to be really, really impressive for what we were trying to do.

So in the early days of this resurgence of deep learning, a lot of the activity was centered on restricted Boltzmann MA and then Deborah machines. There's still a lot of exciting research there being done, including some in your group. But what's happening with Boltzmann machines and restricted Boltzmann machines?

Yeah, that's a very good question. I think that in the early days, the way that we were using restricted Boltzmann machines is you sort of can imagine training a stack of these restricted Boltzmann machines that would allow you to learn effectively one layer at a time. And, and there's a good theory behind, you know, when you add a particular layer, it improves the variational bound and so forth under certain conditions. So there was a theoretical justification and these models were working quite well in terms of being able to pre train these systems. And then around 2009, 2010, once the compute started showing up, you know, GPUs, then a lot of us started realizing that actually directly optimizing these deep neural networks was, you know, was giving similar results or even better results.

So just standard back prop without the pre training or restrictive ultimate machine.

That's right. That's right. And that's sort of over, you know, three or four years. And it was exciting to the whole community because people thought that, wow, you can actually train these deep models using these pre training mechanisms. And then, you know, with more compute, people started realizing that you can just basically do standard back propagation, something that we couldn't do back in 2005 or, you know, 2004, because it would take us months to do it on CPUs. And so that was a big change. The other thing that I think that we haven't really figured out what to do with bolt machines and deep Boltzmann machines, I believe they're very powerful models because you can think of them as generative models. They try to model complex distributions in the data. But when we start looking at learning algorithms, learning algorithms right now they require using market chain, Monte Carlo and variational learning and such, which is not as scalable as back propagation algorithms. So we yet have to figure out more efficient ways of training these models. And also the use of convolution, it's something that's fairly difficult to integrate into these models. I remember some of your work on using probabilistic max pooling for sort of building these generative models of different objects and using these ideas of convolution was also very, very exciting. But at the same time it's still extremely hard to train these models.

Honlak Lee's work.

Yes, Honlak Lee's work. Right. And so we still have to figure out what are on the other side. Some of the recent work using variational encoders, for example, which could be viewed as directed versions of Boltzmann machines, we have figured out ways of training these models, was a work by Max Welling and Dirt Kingma on using reparametrization tricks. And now we can use backpropagation algorithm within the stochastic system, which is driving a lot of progress right now. But we haven't quite figured out how to do that in the case of Boltzmann machines.

So that's actually a very interesting perspective I actually wasn't aware of, which is that in an earlier era where computers were slower, that the rbm, uh, pre training was really important. It was only faster computation that, that, that drove switching to standard back prop, um, you know, in terms of the evolution of communities thinking in deep learning. Another topic that I know we spent a lot of time thinking about is the um, generative unsupervised versus supervised approaches. Do you want to share a bit about how your thinking about that has evolved over time?

Yeah, I think that's a, that's a really, I feel like it's a very important topic, particularly if we think about unsupervised or semi supervised or generative generative models. Because to some extent a lot of successes that we've seen very recently is due to supervised learning. And back in the early days, unsupervised learning was primarily viewed as unsupervised pre training because we didn't know how to train these multi layer systems. And even today if you're working in a settings where you have lots and lots of unlabeled data and a small fraction of labeled examples, these unsupervised pre training models, building these generative models can help for supervised tasks. I think that A lot of us in the community, it's kind of the belief when I started doing my PhD was all about generative models and trying to learn these stacks of models because that was the only way for us to train these systems. Today there is a lot of work right now on generative modeling. You know, if you look at generative adversarial networks, if you look at variational turn cores, deep energy models is something that my lab is working on right now as well. I think it's very exciting research, but we haven't, perhaps we haven't quite figured it out. Again, for many of you who are thinking about getting in the deep learning field, this is one area that I think we will make a lot of progress in hopefully in the near future.

So unsupervised learning.

Unsupervised learning, right. Or maybe you can think of it as unsupervised learning or semi supervised learning where you have, I give you some hints or some examples of what different things mean and I throw you lots and lots of unlabeled data.

So that was actually a very important insight that in an earlier era of deep learning where computers were just slower, the restricted Boltzmann machine and debosement machine that was needed for initializing the neural network weights, but as computers got faster, straight backprop then start to work much better. So you know, one other topic that I know you've spent a lot of time thinking about is the supervised learning versus generative models, unsupervised learning approaches. So how has your. Tell me a bit about how your thinking on, on that debate has evolved over time.

I think that we all believe that we should be able to make progress there. It's just, you know, all the work on Bolston machines, variational two encoders, GANs, you can think a lot of these models as generative models, but we haven't quite figured out how to really make them work and how can you make use of large volumes. And even for I see a lot of in IT sector companies have lots and lots of data, lots of unlabeled data. There's lots of efforts for going through annotations because that's the only way for us to make progress right now. And it seems like we should be able to make use of unlabeled data because it's just abundance of it and we haven't quite figured out how to do that yet.

So you mentioned for people wanting to enter deep learning research, unsupervised learning is an exciting area. Today there are a lot of people wanting to enter deep learning, either research or applied work. So for this global community, either research or applied work, what advice would you have?

Yes, I think that one of the key advices I think I should give is people entering that field. I would encourage them to just try different things and not be afraid to try new things and not be afraid to try to innovate. I can give you one example which is when I was a graduate student, you know, we were looking at neural nets and these are highly non convex systems that are hard to optimize. And I remember talking to my friends within the optimization community and the feedback was always that well, there is no way you can solve these problems because these are non convicts. We don't understand optimization. How could you ever even do that compared to doing convex optimization? And it was surprising because in our lab we never really cared that much about those specific problems. We just were thinking about how can we optimize and whether we can get interesting results. And that effectively was driving the community. So we were not scared, maybe to some extent because we didn't, maybe because we were lacking actually the theory behind optimization. But I would encourage people to just try and not be afraid to try to tackle hard problems.

Yeah, and I remember you once said don't learn to code just in the high level deep learning frameworks, but actually understand.

Yes, that's right. I think that it's one of the things that I try to do when I teach a deep learning class is one of the homeworks I'm asking people to actually code back propagation algorithm for convolutional neural networks. And it's painful, but at the same time if you do it once you really understand how these systems operate and how they work and how you can efficiently implement them on gpu. And I think it's important for you to, when you go into research or industry, you have a really good understanding of what these systems are doing. So it's important, I think since you.

Have both academic experience as a professor and corporate experience, I'm curious if someone wants to enter deep learning, what are their pros and cons of doing a PhD versus joining a company?

Yeah, I think that's, that's actually a very good question. In my particular lab I have a mix of students. Some students want to go and take an academic route, some students want to go and take an industry route. And it's becoming very challenging because you can do amazing research in industry and you can also do amazing research in academia. But in terms of pros and cons, in academia, I feel like you have more freedom to work on long term problems or if you think about some crazy problem, you can work on it. So you have a little bit more freedom. At the same time, the research that you do in industry is also very exciting because in many cases you can, with your research, you can impact millions of users if you develop a core AI technology. And obviously within the industry you have much more resources in terms of compute and be able to do really amazing things. So there are pluses and minuses. It really depends on what you want to do. And right now it's a very interesting environment where academics move to industry and then folks from industry move to academia, but not as much. And so it's, you know, it's very exciting times.

It sounds like academic machine learning is great and corporate machine learning is great and the most important thing is just jump in, right? Either one, just jump in.

So it really depends on your preferences because you can do amazing research in either place.

So you've mentioned unsupervised learning as one exciting frontier for research. Are there other areas that you consider exciting frontiers for research?

Yeah, absolutely. I think that what I see in our community right now, particularly deep learning community, is there are a few trends. One particular area that I think is really exciting is the area of deep reinforcement learning because we were able to figure out how we can train agents in virtual worlds. And this is something that in just the last couple of years to see a lot, a lot of progress of how can we scale these systems, how can we develop new algorithms, how can we get agents to communicate to each other, with each other? And I think that that area is in general, the settings where you're interacting with the environment is super exciting. The other area that I think is really exciting as well is the area of reasoning and natural language understanding. So can we build dialogue based systems? Can we build systems that can reason, that can read text and be able to answer questions intelligently? I think this is something that a lot of research is focusing on right now. And then there's another sort of sub area also is this area of being able to learn from fewer examples. So typically, you know, people think of it as one shot learning or transfer learning. A setting where, you know, you, you learn something about the world and I throw you a new task at you and you can solve this task very quickly, much like humans do, without requiring lots and lots of labeled examples. And so this is something that a lot of us in the community are trying to, to figure out how we can do that. And how can we sort of come closer to human like learning abilities?

Thank you Russ for sharing all the comments and insights. It was especially interesting hearing the story of your early days doing this.

Thanks Andrew. Thanks for having me.