deeplearning.ai's Heroes of Deep Learning: Ian Goodfellow

Hi Yan, thanks a lot for joining us today.

Thank you for inviting me, Andrew. I'm glad to be here.

Cool. Today you're one of the world's most visible deep learning researchers. Let's ask you to share a bit about your personal story. So how do you end up doing this work that you now do?

Yeah, that sounds great. I guess I first became interested in machine learning right before I met you. Actually. I'd been working on neuroscience and my undergraduate advisor, Jerry Kane at Stanford encouraged me to take your Intro to AI class.

Oh, I didn't know that.

Okay. So I had always thought that AI was a good idea, but that in practice the main thing I knew that was happening was like game AI where people have a lot of hard coded rules for non player characters in games to say different scripted lines at different points in time. And then when I took your Intro to AI class and you covered topics like linear regression and the bias and various variance decomposition of the error of linear regression, I started to realize that this was a real science and I could actually have a scientific career in AI rather than neuroscience.

I see. Great. And then what happened?

Well, I came back and ita'd your course later. So a really big turning point for me was While I was T.A. that course, one of the students, my friend Ethan Dreyfuss, got interested in Jeff Hinton's Deep Beliefnet paper and the two of us ended up building one of the first GPU CUDA based machines at Stanford in order to run Boltzmann machines in our spare time over winter break. And at that point I started to have a very strong intuition that deep learning was the way to go in the future, that a lot of the other algorithms that I was working with, like support vector machines, didn't seem to have the right asymptotics, that you add more training data and they get slower or for the same amount of training data, it's hard to make them perform a lot better by changing other settings. And at that point I, I started to focus on deep learning as much as possible.

And then in the. And, and I remember Rajet Rayner's very old GPU paper.

Right.

Acknowledges you for having done a lot.

Yeah, yeah, that was written using some of the machines that we built. The first machine I built was just something that Ethan and I built, Ethan's mom's house, with our own money. And then later we used lab money to build the first two or three for the Stanford lab.

Wow, that's great. I never knew that story. That's great. And Then today, one of the, you know, things that's really taken the deep learning world by storm is your invention of GANs. So how did you come up with that?

I've been studying generative models for a long time. So GANS are a way of doing generative modeling where you have a lot of training data and you'd like to learn to produce more examples that resemble the training data. But. But they're imaginary. They've never been seen exactly in that form before. There were several other ways of doing generative models that had been popular for several years. Before I had the idea for GANS and after I'd been working on all those other methods throughout Most of my PhD, I knew a lot about the advantages and disadvantages of all the other frameworks like Bolton machines and sparse coding and all the other approaches that had been really popular for years. I was looking for something that would avoid all of those disadvantages at the same time. And then finally, when I was arguing about generative models with my friends in a bar, something clicked into place and I started telling them, you need to do this, this and this, and I swear it'll work. And my friends didn't believe me that it would work. I was supposed to be writing the deep learning textbook at the time, but I believed strongly enough that it would work that I went home and coded it up the same night and it worked.

So it took you one evening to implement the first version of ganzling?

I implemented it around midnight after going home from the bar where my friend had his going away party. I see. And the first version of it worked, which is very, very fortunate. I didn't have to search for hyperparameters or anything.

There was a story I read somewhere where you had a near death experience and that reaffirmed your commitment to AI. Tell me that story.

Yeah, I wasn't actually near death, but I briefly thought that I was. I had a very bad headache and some of the doctors thought that I might have a brain hemorrhage. And during the time that I was waiting for my MRI results to find out whether I had a brain hemorrhage or not, I realized that most of the thoughts I was having were about making sure that other people would eventually try out the research ideas that I had at the time. In retrospect, they're all pretty silly research ideas, but at that point I realized that this was actually one of my highest priorities in life, was carrying out my machine learning research work.

I see. Yeah. That's great that when you thought you might be dying soon. You're just thinking how to get the research done. Yeah, yeah, that, that, that, that, that's commitment.

Yeah, yeah.

So today you're still at the center of a lot of the activities with GANs with generative adversarial networks. So tell me how you see the future of GANs right now GANs are.

Used for a lot of different things like semi supervised learning, generating training data for other models, and even simulating scientific experiments. In principle, all of these things could be done by other kinds of generative models. So I think that GANs are at an important crossroads right now. Right now they work well some of the time, but it can be more of an art than a science to really bring that performance out of them. That's more or less how people felt about deep learning in general 10 years ago. And back then we were using deep belief networks with Bolton machines as the building blocks. They were very, very finicky. Over time we switched to things like rectified linear units and batch normalization and deep learning became a lot more reliable. If we can make GANs become as reliable as deep learning has become, then I think we'll keep seeing gans used in all the places they're used today with much greater success. If we aren't able to figure out how to stabilize GANs, then I think their main contribution to the history of deep learning is that they will have shown people how to do all these tasks that involve generative modeling. And eventually we'll, we'll replace them with other forms of generative models. So I spend maybe about 40% of my time right now working on stabilizing GANs.

I see. Cool. Oh, and so, so just as a lot of people that joined deep learning about 10 years ago, such as yourself, wound up being part of, maybe the people that join GANs today, if it works out, could end up the early pioneers.

Yeah, a lot of people already are early pioneers of GANs. And I think if you wanted to give any kind of history of GANS so far, you'd really need to mention other groups like Indico and Facebook and Berkeley for all the different things that they've done.

So in addition to all your research, you also co authored a book on deep learning. How is that going?

That's right. With Yoshua Bengio and Aaron Courville, who were my PhD co advisors. We wrote the first textbook on the modern version of deep learning and that has been very popular both in the English edition and the Chinese edition. We've sold about, I think around 70,000 copies total between those two languages. And I've had a lot of feedback from students who say that they've learned a lot from it. One thing that we did a little bit differently than some other books is we start with a very focused introduction to the kind of math that you need to do deep learning. I think one thing that I got from your courses at Stanford is that linear algebra and probability are very important that people get excited about the machine learning algorithms. But if you want to be a really excellent practitioner, you've got to master the basic math that underlies the whole approach in the first place. So we make sure to give a very focused presentation of the math basics at the start of the book. That way you don't need to go ahead and learn all of linear algebra, but you can get a very quick crash course in the pieces of linear algebra that are the most useful for deep learning.

So even someone whose map, you know, is a little shaky or haven't seen the map for a few years would be to start from the beginning of your book and get that background and get into deep learning.

All of the facts that you would need to know are there. It would definitely take some focused effort to practice at making use of them. If someone's really afraid of math, it might be a bit of a painful experience. But if you're ready for the learning experience and you believe you can master it, I think all the tools that you need are there.

Um, as someone that's worked in deep learning for a long time, um, I'd be curious if you look back over the years. Tell me a bit. How about how your thinking of AI and of deep learning has evolved over the years.

Ten years ago, I felt like as a community, the biggest challenge in machine learning was just how to get it working for AI related tasks at all. We had really good tools that we could use for simpler tasks where we wanted to recognize patterns in hand extractor features, where a human designer could do a lot of the work by creating those features and then hand it off to the computer. And that was really good for different things like predicting which ads a user would click on, or different kinds of basic scientific analysis. But we really struggled to do anything involving millions of pixels in an image or a raw audio waveform, where the system had to build all of its understanding from scratch. We finally got over that hurdle really thoroughly maybe five years ago. And now we're at a point where there are so many different paths open that someone who wants to get involved in AI, maybe the hardest problem they Face is choosing which path they want to go down. Do you want to make reinforcement learning work as well as supervised learning works? Do you want to make unsupervised learning work as well as supervised learning works? Do you want to make sure that machine learning algorithms are fair and don't reflect biases that we prefer to avoid? Do you want to make sure that the societal issues surrounding AI work out well, that we are able to make sure that AI benefits everyone rather than causing social upheaval and trouble with loss of jobs? I think right now there's just really an amazing amount of different things that can be done, both to prevent downsides from AI, but also to make sure that we leverage all of the upsides that it offers us.

And so today there are a lot of people wanting to get into AI. So what advice would you have for someone like that?

I think a lot of people that want to get into AI start thinking that they absolutely need to get a PhD or some other kind of credential like that. I don't think that's actually a requirement anymore. One way that you could get a lot of attention is to write good code and put it on GitHub. If you have an interesting project that solves a problem that someone working at a top lab wanted to solve, once they find your GitHub repository, they'll come find you and ask you to come work there. A lot of the people that I've hired or recruited at OpenAI last year or at Google this year, I first became interested in working with them because of something that I saw that they'd released in open source form on the Internet. Writing papers and Putting them on arXiv can also be good. A lot of the time it's harder to reach the point where you have something polished enough to really be a new academic contribution to the scientific literature. But you can often get to the point of having a useful software product much earlier.

So. So, you know, read your book, practice the materials, and post on GitHub and maybe on Archive.

I think if you, if you learn by reading the book, it's really important to also work on a project at the same time to either choose some way of applying machine learning to an area that you're already interested in. Like if you're a field biologist and you want to get into deep learning, maybe you could use it to identify birds. Or if you don't have an idea for how you'd like to use machine learning in your own life, you could pick something like making a street view, house numbers Classifier where all the data sets are set up to make it very straightforward for you. That way you get to exercise all of the basic skills while you read the book or while you watch Coursera videos that explain the concepts to you.

So over the last couple years, I've also seen you do more and more work on adversarial examples. Tell us a bit about that.

Yeah, I think adversarial examples are the beginning of a new field that I call machine learning security. In the past, we've seen computer security issues where attackers could fool a computer into running the wrong code. That's called application level security. And there's been attacks where people can fool a computer into believing that messages on a network come from somebody that is not actually who they say they are. That's called network level security. Now we're starting to see that you can also fool machine learning algorithms into doing things they shouldn't, even if the program running the machine learning algorithm is running the correct code, even if the program running the machine learning algorithm knows who all the messages on the network really came from. And I think it's important to build security into any new technology. Near the start of its development, we found that it's very hard to build a working system first and then add security later. So I am really excited about the idea that if we dive in and start anticipating security problems with machine learning now, we can make sure that these algorithms are secure from the start instead of trying to patch it in retroactively years later.

Thank you. That was great. There was a lot about your story that I thought was fascinating and that despite having known you for years, I didn't actually know. So thank you for sharing all that.

Oh, very welcome. Thank you for inviting me. It was a great chat.

Thank you.

Very welcome.