Interview between Andrew Ng and Quoc Le (Google Brain researcher) covering Quoc Le's path into AI (Vietnam → Australia → Germany → Stanford → Google Brain), key research contributions (Google Brain large-scale unsupervised visual experiment that discovered face/cat-sensitive neurons; sequence-to-sequence neural machine translation with LSTMs and later scaling; Meena conversational chatbot using Transformers), reflections on why scale and training matter, interest in generative models and factuality, and career advice (patience, start small, reproduce papers, value of naivety).
Chapters
Hey, everyone. Welcome back to this interview series. I'm delighted to have with us Kwok Le, who is a researcher at Google Brain. Um, I had the pleasure of working with him at Stanford University many years ago, and Kwok and I were chatting just now. Fun fact, Kwok was the very first intern I had recruited into Google Brain when I was leading it back in the early days. Ji Chen Yam was the second intern I ever recruited into Google Brain and I think Jeff Hinton was the third intern I ever recruited. So I think Kwok was in good company. Since then, Kwok has done some of the most influential work in nlp and I'm delighted to have him here today to tell us about some of his experiences. Thanks for joining us, Kwong.
Yeah, thank you, Andrew, for having me in the interview series and it's a pleasure to be here talking to you.
So today, Kwok, you are known widely as one of the most influential NLP and deep learning researchers. And your journey has been a complicated one. I think you had started off going to school in Vietnam and then went to Australia and then Germany and then now the United States. So tell us more about your journey. Getting into AI.
Yeah, so when I was like a high school student, I was fascinated by AI and I ended up reading a lot of books and even program some simple AI programs. And then I got a scholarship from Australia for my undergrad in the Australian National University. And in my second year of my undergrad, I got a little bit bored and I said, you know, maybe I should do something research in AI, because it seems like the faculty there, they have some very amazing faculty. So I contacted Alex Mola, who actually took me as an intern for a summer, and I work on kernel methods with him. And that's the. I think that even though I did some AI, you know, projects before that, that's the first time that I actually learned about machine learning. And I became super fascinated about machine learning and I realized the potential of doing machine learning and learning for AI.
Wait, wait, backing up a bit. I don't think I knew this story. What was the AI project you were coding up in high school that got your scholarship to Australia?
I didn't get a scholarship to Australia because of the AI program, but I programmed it myself to learn more about AI and so on. I was building like a chatbot, you know, like rule based system to actually talk to myself, right? And you know, try to randomize the answer and so on. Try to see how I can fool my friends because I read about the Turing Test and so on. And I was fascinated about it. But I didn't win the scholarship because of that program. It was too simple to flow it. But I, you know, I, because when I program it, know how hard it is to write a computer program to do, to actually be intelligent.
And I think it's really cool that you started out that way because I think if there's someone listening to this interview series now that is coding up, you know, some slightly broken chatbot that kind of works, doesn't work and finds it hard, right? They may be taking their first steps to an illustrious career in AI. Much like you were once coding out like a so so chatbot. I find it inspiring that we all start from programs that don't quite work as well as we wish. But that could be a wonderful start.
Actually, much later, many years later, I actually had a conversation with you and turns out you also did something like this as well in the past. But I was fascinated, I was always fascinated about if you ever build an AI program, can I talk to it, right? Can I talk to the program and hear about what it thinks? But anyway, going back to the journey, I did my summer project with Alex Mola on kernel methods machine learning, and I realized the potential of using machine learning to build AI. And then after that I graduated from, kept on doing research with him for a while. And then after I graduated, he has a friend in Germany and I went to German, you know, Bernard Schonkopp hosted me in Germany. And when I was in Germany, I was also doing machine learning research. And near the machine learning research, there's like a neuroscience center that they were trying to think about, trying to understand the brain and so on. So I became very fascinated about that as well. Then I listened to one of your talks at Neurips that year that you were talking about using machine learning for AI. By the way, at that point in time, most people think about machine learning as pattern recognition, just doing machine learning for machine learning, trying to classify things. But when I heard about machine learning to do AI, it really resonated with me. So I applied to the Stanford PhD program to do the PhD. And then, you know, I ended up going to Stanford and then doing the PhD program, the PhD degree with you. So that's around 2007, and then around 2010, 2011, you, you told me that you were founding and creating a project at Google to scale up deep learning research and make deep learning like 100 or thousand times bigger. When I heard about that, I said, oh, that must be the future, because in the Neural nets I trained. No matter what I did, the only thing that makes a big difference is giving it more data and more compute. So it really resonated with me.
I'm glad you thought it was the future. A lot of my friends at the time were telling me it was a terrible idea to do this crazy Google Brain thing. So I'm glad you thought it was reasonable. But actually I think you and I, we had the privilege at Stanford of seeing some of the early results that Adam Coates was generating. And Adam Coates has shown using small scale resources and scaling up the neural networks really work. So I think it made it easier.
Yeah, I totally agree. That figure that Adam Coates did, I think, I mean that's basically changing a lot of the things that I did, but I think probably a lot of the things in the field did too, but really influenced me into, you know, joining the project. Of course I then I went to Google Brain and I think you and I was an intern like you said. I was the first intern of the project and that's around 2011, around 2000 I think. I remember that around April 2011. And then I stayed since with the project and you know, brain has grown, grown from, you know, two, four, five people that you had at the beginning and now to, you know, much larger teams and now I do research, still continue to do research in deep learning and AI and Yeah, yeah, yeah, Google.
Brain was incredibly lucky to have you join and I think one of the first home runs was that, you know, Google Cat project which you were the lead engineer and the lead author on, if you remember that.
Yeah, yeah, yeah, yeah, yeah, I remember that. So back when we were at Stanford, I was, you know, playing around with this method called Autoencoder, basically given the input image in and trying to reconstruct the image. And then if you apply some kind of sparsity, you start having some at the low level of the neural network, you start seeing some kind of like edge filters. And edge filter is very useful for computer vision. And then there's this influential result by Mahong Lak Lee, which is our lab mate and I think in the project he will hit it with you. Is that like if you do this in a deeper neural network, you start having more sophisticated features. And for me at that point in time, you know, learning feature representation is very, very interesting. And our taught ethics exercise, this is our, it's not me, but our thought exercise, you got a lot of credit in this, so I have to say it. But our thought exercise was actually can you make this 100 or thousand times bigger and learn even more sophisticated features. We don't know what they're going to end up doing. Like these networks. Right. And then we set out doing this project. We scale it up from a single machine to 16,000 machines. I remember that. And we trained it on YouTube because I feel like my question was, what is the biggest image data set that we have access in the world at that moment? And it turns out that YouTube images. So we trained the neural network for like a week or so, the sparse autoencoder. And you know, with a lot of poking we found some hidden, some neuron in the network that is very sensitive to faces and cats, the typical things that people expect on the Internet. And yeah, and one of the most intriguing result was that it's actually faced, surprisingly the network found cats on the Internet.
I still remember, I think we're both in the office that day. The team was so much smaller and you grabbed me and said hey Andrew, come to my computer and take a look. And I walked in and said whoa. And that was the, the cat thing that, that you had discovered from unsupervised venue.
Yeah, yeah.
So, and, and I think, you know, and I think one, one piece of early deep learning history is, I think almost all of us maybe overestimated the short term impact of unsupervised learning and didn't realize that supervised learning will be where a lot of the action, you know, at least in the early days would be. Um, although I find it interesting that one place where unsupervised learning has really taken off and is showing wonderful commercial results is in nlp. Yeah, yeah, yeah, yeah. And in fact I still remember when, you know, there was one day in the office where we were in Google Rain where you walked over a tat show there and you said, hey Andrew, come to the computer and take a look and walked over and there, you know, you gotten this cat to appear on your monitor from unsupervised learning. And I think that iconic, slightly blurry Google Cat wound up being one of the iconic images of unsupervised learning for that time. So I think team couldn't have done that without you. Shortly after, one of the other huge hits that you worked on was the sequence to sequence model. And I think that was another piece of work that changed the trajectory of nlp. Tell us a bit more about that.
The first version of sequence to sequence is actually a lot of people did not know, but there's this idea called word to world is translating word from one language to another. And the reason why we did this is the following. So back in around 2012, 2013, so suddenly there's this wave of new world vector methods and pioneered by a guy called Thomas Mikulov and He was developing World2Vec, and it trains world vectors very quickly. And it shows this amazing ability of if you take king minus green, then the vector is similar to man minus woman, for example, and basically solving analogy. So the next question we ask is that is there any similar structure between the two languages? So for example, can you train word vectors in Spanish and then also train word vectors in English? And can you try to align them a little bit? For example, can you learn that certain words in one language align with another word? And we did this thing and it turns out it works. So you can only have, let's say, five words or you know, 100 words in popular words in English and you know, the corresponding translation in Spanish. And then you learn the rotation matrix and then you can actually map. Now you can start translating words from one language to the other without knowing too much about the language. So that, you know, this word to word translation gave me the feeling that, oh, maybe we should not do this word to word. Maybe we should do sentence to sentence. And then we say, okay, so how do we do sentence to sentence? And it turns out that it wasn't easy conceptually at the beginning for us. So we were thinking other ways. For example, let's try to and read in the input sequence the input sentence and then try to predict, you know, what the first word looks like. And then trying to predict what's the second world look like without knowing the first word. For example, it's like predicting the first word, second world, third word, etc. Without knowing their relation. And then I said that, oh, maybe that's a little bit limited. Because if you predict a second world, you should know the first word. Because if you, and if you predict the third word, you should know the second word, etc. Right. So it has to be some, some sort of dependency. So we, we thought about it.
In an English sentence, it, you know, doesn't really make sense to independently predict the first and the second and the third and the fourth words. You should predict the first word and then based on that, figure out what's the second word. And then based on that, only after that, figure out what the third word. And the sentences you output just make a lot more sense when you do it one word at a time rather than try to spew out all 10 words at the same time that may not end up relating to each other.
Yes, yes, that's a good way to explain it. Now it turns out that, so this, you know, like, from the point that we realized this idea to actually making it work, it took like a year. And it was actually very difficult. And one bottleneck about this whole thing is actually training sequence model to do this. And back at the time when I was training this, I trained with a conventional recurrent neural network. But at that point in time, together with me, Ilya Sutkova and Orio Vignal, they all also have very similar ideas of doing this. And Orio Vignal and Ilya Suskava, they know how to train this model, how to get this model to work with lstm. So we met and then we decided that let's try with the lstm. It turns out it gave us really good improvements. But when the first translation, it still looked awful. And it can output certain words that look like the input, but mostly wrong. So the question is, should we continue? And, you know, I was into, you know, trying all sort of ideas and so on, but credit to Ilya that he said maybe we should train this model better. We should really train this model better and train a bigger model, Train bigger model and train it longer, train it better. And credit to him that actually, actually that approach actually turns out to be successful. And then he, we train it, we lower the perplexity slowly, slowly over time. And then we start when after a couple of months, we start seeing better and better translation. Now it's just only one or two words, but like five words, certainly more correct. And we realized that we were onto something.
So it sounds like if I got the history right, you tried it with RNNs, then got better results with LSTMs. And then the last mile was just scaling more data, bigger network, bigger LSTM network. And that wound up being the last several steps to get to that, you know, seminar breakthrough. That redensal and error is.
Yeah, I would say that like, last seems trivial from the perspective of a researcher, but like, I think it makes a huge difference for the project because, you know, had I tried, you know, fancy ideas and, you know, we also like crazy ideas, I think I would not get the thing to work and we would delay the project for much longer. And it's a lot of lessons. Even though the last mile seems to be trivial from the research point of view, it made a huge difference for the project. And had we invested in fancy ideas and not didn't Try scale and didn't try to train the model longer or better. If we invest more on ideas, we never got the model to work and it would delay the project for much longer.
Yeah, doesn't seem trivial to me at all. And I find that when I'm building systems, when I'm in basic research mode, sometimes I hold the data set fixed and change the algorithm.
But.
But when I'm in production, just build a commercial system that works well, sometimes I hold the algorithm fix, you know, pick an architecture that I think is good enough and then spend all my time changing the data. So it's very, very different ways of operating.
Yes, yes, yes.
So the work that you did with Elia and Oriole on sequence, sequence models. Right. This is a seminal piece of work. Really huge impact on the whole field of deep learning. And then almost coming full circle, you built chatbots in high school, and building on the ideas of sequence to sequence, you also built this much more advanced chatbot than your high school version, I imagine, called Meaner. And I remember reading that paper and thinking, wow, some of these outputs actually look pretty cool. Talk a bit about the meaner chatbot project.
Sure, sure. One of my dreams in my high school is that the chatbot that can tell you a joke, you know, it can tell me like a new original joke. And you know, one of the things.
When we're writing the badge every Tuesday because we stand on Wednesday, is my editor in chief and I often end up sitting around brainstorming for five minutes to see what's the joke we want to tell one of the other people. So if you're a chatbot to automate joke telling, it'll make my life easier.
Yeah, that's for sure. Same with me. You know, sometimes I want to give a talk and I always begin the talk with the some joke. And I find it sometimes hard to make the joke as well. But. Yeah, but like when I was in high school, I was thinking, you know, can we come up with a way to create like an original joke? And it turns out if you think about through that exercise, it's actually difficult, more difficult than we can program. So after I worked on the sequence to sequence project, I realized that if you can get in sequence as input and then produce sequences output, maybe one of the biggest application is to train a model can talk to you. Now this idea actually executed better. Who thought about this idea at Google first is not me, it's actually Oreo Vigna. So he actually trained a chatbot that actually worked really well. And we Realized that maybe we should collaborate and then build a chatbot. Cool Chatbot. And we did it in around 2016 or something like that. And it's like a paper on archive. And we show that there's potential of this.
Remember, you do a lot of work within Google to get permissions to use the IT support to support.
Exactly.
Data in order to get into the chatbot.
Yeah, yeah. So back at the time, we were like, we were researchers. We asked ourselves, where's the data set? So one thing, what we did was going after the technical support internally at Google and get a data set from them and then train a model. And then we asked a question like, can you debug this? I lost my password. What should I do? And then it start answering some kind of question. Although. Although it still feels a little bit off, but you start seeing there's something there. Right. There's something there that's really promising. And then I spent.
I just have this very clear recollection. You, me, I think Adam Coates, a bunch of us were having dinner in a restaurant in Palo Alto and you had just gotten permission to use the data and you were very excited over dinner that day. Exactly.
Yeah, yeah.
Collection of that dinner.
Yes, yes. That was when we had a sushi in the Palo Alto restaurant. Yeah, yeah, yeah, I remember we talked about that and how excited you were that, you know, someone worked on Chatbots, because you also worked on Chatbot back in the days. But yeah, and then.
And I think I told you about my high school chatpot attempt was actually more of a prank. Right. I think I told you I wrote a, you know, prank chatbot where I will type, and no matter what I type, the chatbot will print out what is your name? So I would type, you know, quoc space, L, E. And as I type that, the chatbot will print out what space is your name. And the effect was when I hit enter, then it will print out what I actually typed. So to the user, they see, they think I typed what is your name? And then it prints Kwoklur. And so to my friends, I was pranking. It looked magical because I could ask it the question, they'll know the answer. But what was actually happening was I was typing the answer. It just was displaying a pre canned question. So that's impressive. No, AI, just a high school kid prank. Let's see, your meaner was much more interesting. So tell us that story.
I spent one year trying to improve the bot myself and I didn't get very far. And turns out it's very hard to improve after you actually. So if you train multi turn conversation after one or two turns it becomes very difficult. For example, it can answer you the first turn, but the second turn it starts to forget information. It doesn't give you, you know, good answer and so on. So it's pretty much broken. But then by the end of 2016 and 2017, I met this guy at Google. He's an engineer at Google, his name is Daniel. And came and told me and said I also had a dream to be a chatbot. So let's work together to build a chatbot. And when he came in back at that moment, something magical happened which is basically people have developed this transformer architecture and we say, okay, maybe the transformer is more promising because it can understand long range dependency better. So it can see multiple turn better than the lstm. And so we started working on using the transformer to deal with this multi turn conversation. And we train bigger model again longer and using more resources at Google and slowly we start to see some improvement. And one magical moment in my opinion is that I look through the log so we allow some people to chat to it at Google. And there was one magical moment is that we looked through the log and then I saw some joke the bot made and we, and the, it's actually in the paper about, you know, the joke was about, you know, if cows go to Harvard, then horses should go to Hayward. It's a funny dad joke. A dad joke. And then, but, but it's actually a multi tone conversation. It's trying to lead the audience into it. So I thought, and I thought, oh. So I tried to look into the training data to see whether it's actually, does it have anything like Hayward or you know, cows go to Hayward, sorry, horses go to Hayward or not. And we couldn't find it. So it's truly, it's actually trying to understand the concept of jokes, it understands the concept of puns and it actually created a new joke. And I think that's a very magical moment in my opinion. And pretty excited about it, I think.
Yeah, I remember reading that joke when I read the Mina paper. Hey, how do you know if it was true? Understanding the concept of jokes and puns and Harvard versus Hayward versus, you know, if you, if you. Enough monkeys on a typewriter or enough randomness means eventually there will be a funny joke.
Yeah. So we look into the training data very carefully and see, look for the word havard. Right. And see whether that word mentioned anywhere. It mentioned once in the entire training Corpus, but it's not next to like a joke. It's mentioned in a different context. Our conclusion is actually trying to actually understand this concept of a joke. Because the training data set has a lot of jokes. Training in the dataset has a lot of jokes as well. Bumps. There's a lot of bumps. But that particular example is actually novel. We could not find anywhere in the training data.
So you've been at the forefront of, you know, NLP research for many years, and then we continue to see breakthroughs in NLP on a regular basis. So I'm curious, looking to the future, what are you most excited about in terms of things yet to come in nlp?
Oh, yeah, I think I'm most in nlp. I'm mostly excited about generative models. I think currently a lot of NLP is basically doing traditional NLP where you can classify the sentiments of the sentence or you can do name entity recognition on the sentence. But I'm more excited about the capability of generation. Like, can you generate a new book that consumed by humans and teach people a new concept? Can we get to that point or maybe can it help a director or screenplay writer to come up with a better movie plot? There's so much potential in using these technology for generation, in my opinion.
Since the breakthroughs of transformer model and the several flavors of Transformers, one huge vector of progress for the generative models has been scale, right? Scale of data and especially compute. Other than scale, what do you think are the important vectors of progress for generative models?
Okay, I think. True understanding. I think right now we look into, for example, text that generates. There's still things that a little bit off it made up stuff, right? Like there's some facts that it made up. So the question is, can we get the bot to really have common sense understanding and generating more factually correct output? I think that would be a big thing. I think that would be a big factor of progress.
I want to dig a little bit further into that because I think a lot of researchers, including me, sometimes have talked about computers understanding images or understanding language. And common sense is a concept that philosophers have debated for, I think over 2000 years now. So when you talk about NLP understanding or common sense, are these scientific concepts that are measurable or are these philosophical concepts that you kind of feel like it understands it? But how do you approach these questions? Understanding common sense?
I think it's measurable. So in some sense you can ask, for example, you can, for example, you can create, for example, GPT or any language model. You can give a prompt about, let's say, you know, talking about the GDP in the world. Let's talk about GDP in the world. And can you look at the GDP and you compare with Wikipedia and can you say they are, they're matching and that's measurable? Or you can start talking about movies in 2020 and can it understand that these are the movies? I think it's measurable.
Do you have a favorite set of benchmarks? Like there's like a common sense QA is one data set. Do you have a favorite benchmark for measuring understanding or common sense? I thought common sense is a bold name to say this is our measure of common sense. It's a bold name for data science.
Maybe we can, instead of talking about common sense, we can just talk about maybe factuality a little bit. Just factual knowledge. Just generate some statements that are actually factually correct. For example, how always is Barack Obama and so on. It can generate correctly. And from that perspective, I think it's easy to create a data set for that. We haven't had that data set yet, but I think it will be created. I think that people are in other than AI, they created common sense data sets to measure the performance of genetic models. I'm not expert in this field, but I think my feeling is that when I read a lot of these outputs, I still see things that are actually factually not correct. And I think see things a little bit off. And I think addressing that issue can be quite important.
So, like a test set measuring factual correctness would be a good way to help the view move forward. Yeah. In your time at Google, you've mentored a lot of younger engineers. And even at your time at Stanford, you mentored a lot of younger students. So today there are a lot of people wanting to break into AI or they want to advance or to grow their career in AI. What advice do you have to someone wanting to build their career in AI?
Yeah. Okay. So first of all, I have to say that I don't have true meta advice. I don't like the concept of meta advice for a lot of people because I think food for me is poison for other people. We are living in a different world now. But I say, you know, looking through my journey, I say that I started out from something very, very different. Right. I came from different culture, different background. So maybe one piece of advice is that, you know, like career as a whole is taking a long time to make really significant progress. I never can imagine that I make a big progress. I make a good contribution for Translation, for example. So my thinking is that maybe you can start quite low, but maybe with hard work and dedication over time you can do good work and make impact for the field and so be patient, really. So for me, that journey has taken me 15 years also already, from the point that I started and then until I become a more senior researcher at Google. So be patient because it takes time to do impactful work. That's number one. And number two is that I think naivety can be good. A lot of people think that they have to read a lot and understand everything. I think naivety sometimes can be in your favor. For example, for example, in around 2014, when I was doing this sequence to sequence 2013, I tried to reimplement phrase based and neural machine translation. And I was just a bad programmer, I didn't know anything about phrase based maneuver machine translation. I was bad. I spent two months implementing it, I couldn't do it. And because of that, because of that frustration, I, I say, okay, maybe I should do something here. I should do end to end learning instead. But imagine if I was a good programmer and I knew a lot about phrase based machine translation. I would program it successfully and then I would not develop a sequence to sequence. I would not participate in a different project like that. So maybe naivete can be good.
Yeah, I think what you said about patience is great advice and sometimes it does take some time, years to become really good at AI and it can be a long journey, but extremely rewarding journey. So I'm curious, what advice do you have for someone that is learning about AI, learning about machine learning, growing their career, and in the course of that process, how can someone know if they're making progress and how can someone know if they're doing the right things? To be advancing at the right speed.
We need to identify a data set, identify a task where you can see progress more quickly and maybe sometimes you don't get success on that. Then try to understand why and try to talk to maybe contact more senior researchers and try to figure out all the senior engineers and trying to figure out what went wrong. I mean, if I were to start AI today in career in AI as a researcher, let's say, then I would probably, maybe the starting point would be I would try to replicate some of my favorite AI research papers and trying to find, read the paper, try to understand it, reproduce the result, compare against their GitHub, and try to, you know, to get into the habit that I can implement quickly and, you know, my code can reproduce their result quickly and then you Know, if I make progress in that easy phase, then I can start thinking about how could I contribute, how could I get better at developing my own ideas? So that's what I do. And then I would start, in other words, I would start with something easier first. Try to get something that easier. It's like learning to swim. You don't want to go to the ocean to learn to swim. You want to learn to swim in a pool first and then do something quite easy that you get happy doing it, and then slowly move yourself towards more and more difficult tasks. And for example, creating your new idea, writing your own paper, that's difficult. But I think reproducing an existing paper and getting a result on MNIST or cifa, that seems easy enough to do. So maybe start with that first. Start with something simple. Yeah.
Kind of like we use curriculum learning in training neural networks as well. Thanks, kwa. This has been great. Before we wrap up, do you have any final words or any final thoughts for the, for the viewers watching this?
Oh, I can say some of my final words. I say, you know, I accepted the interview because I want to see Andrew and I want to thank, you know, Andrew was a spectacular educator, I have to say. So. I remember in 2017, I came to, you know, I was already a researcher. I wrote papers and so on. I came to Stanford, started my PhD. I came to a lecture hall by you, the CS229 lecture. And I literally had goosebumps because your lecture was so good. It was so. It was like another level. And I learned a lot. I think I learned a lot from, from your class and, you know, from your original thinking as well. Thank you for your contribution in educating the community about deep learning. I think that's, that's going to be very, very impactful for the field as well, because we need more, you know, more people to move the field forward as well.
Yeah, thanks. Thanks, Hua. Well, I wasn't expecting to say that. It's very, very, very, very kind of you. It means, means a lot.
Yeah. Thank you.
And I think. Thank, thanks. Thanks for being with me on this interview series. It's wonderful to see all the tremendous contributions you're making to NLP deep learning. It was great chatting, as always.
Yeah. Yeah. Thank you. For more interviews with NLP thought leaders, check out the DeepLearning AI YouTube channel or enroll in the NLP specialization on Coursera.
