Break into NLP hosted by deeplearning.ai

Hi everyone and welcome. My name is Ryan Keenan and I'm the director of products here at DeepLearning AI. We really appreciate you taking some time for this event today and we hope that you and all your families are doing well. These have been difficult times for the last several months. We know that wherever you are in the world, and we have something like people joining us from something like over 140 countries today, we know that wherever you are in the world, it's been a difficult several months and it's not over yet. So we hope that you're doing well and we hope that maybe you've had something good come out of this, like more time with family or a new opportunity or something else. But today we are gathered to celebrate the launch of our third course in the Natural Language Processing specialization. And we've assembled a panel of experts to talk about nlp, about the work that they're doing, about the future of the field, and give some perspectives. So NLP is really all about giving machines the ability to interpret and manipulate human language. And so as all sorts of different fields in AI continue to expand and make significant contributions to industry, the demand for people with skills in these areas is only growing. So we're really excited today to launch this course. Our mission at DeepLearning AI is to create great courses, really, and to make world class AI education accessible to everyone. So our goal is really to empower our community of developers and business leaders and people changing their career paths, all of you, to be part of this AI transformation that's happening all over the world. So these courses are created by experts from industry and academia. And course three is sequence models in nlp. It just went live a few hours ago. And so this is a really exciting course in the sense that with courses one and two, the skills that you built, if you were a learner in those courses were sort of foundational skills, the NLP basic toolkit. And then now with this course, you're moving into more sort of real world applications, so more powerful models and applications. So super exciting course. And today's panel of experts that we've lined up have a lot of interesting input on the field and on all these different techniques in nlp. We've also lined up a sneak peek of the courses and we'll have a course demo at the end, so stick around for that. Today our agenda will be opening up with a keynote by Lucas Kaiser, who's one of the instructors in the NLP courses, as well as a scientist at Google Brain. Following that, we'll have a talk by Andrew, who's the founder of Deep Learning and deeplearning AI and then we'll have a panel discussion where we'll bring in, in addition to Lucas and Andrew, we'll be joined by Kenneth Church from Baidu usa, Professor Marty Hearst from UC Berkeley, and Eunice Mori from Stanford, who's also one of the instructors in the NLP course. After that we'll do a Q and A session and, and we'll select, we'll do a panel discussion and then we'll have a Q and A session where we select some of the most upvoted questions from our NLP learner Slack community, which is the Slack community you can join once you enroll in the specialization on Coursera. We've got several thousand of you in there already. And then following all that, we'll do the Course 3 demo. So without further ado, I'd like to welcome our first speaker, Lucas Kaiser, who's as I said, one of the instructors in our NLP specialization. Lucas is the CO author of TensorFlow as well as the TensorTO tensor and tracks libraries and he's a co author of the Transformer paper which has really transformed the field of NLP and deep learning in general. He's a staff research scientist at Google Brain, as I mentioned. And so it's a great honor and a pleasure to welcome you to this forum today. Lucas, I'll turn it over to you now.

Thank you very much, Ryan. It's great to be able to see you. I want to tell you a little bit about how I came to deep learning in nlp. Launching this course is a very personal thing for me as well because I've been working for the last number of years on using deep learning in NLP and making it accessible to people. And it's been a wild journey. So we started the work, I mean around the year 2014. People in the NLP community did not believe very much in deep learning, of course, not all of them. Some did, but many did not because the best performing methods were still not deep learning based. They were based on probabilistic models. You've learned about them in the earlier courses and they gave good results. For example, machine translation, which is the field I've been working on mostly was the performance there is measured by something called the BLEU score and it's the higher the better. And around the year 2013 14, the best Bleuscore was obtained by phrase based systems on this English German benchmark, which is a standard data set. It was about 21. And it was considered quite good. But for comparison, like human translator score would be around 30. And people were thinking, okay, these deep learning systems that came around then they were maybe got 19, maybe managed to get 20, but they were very large. And we managed to train just larger networks. And these networks you will learn about LSTMS in the course, they're not complicated, but the larger you make them, the better they work. And around 2016 they got really large and really good. We got 25 blue, which is when the Google Translate launched a neural network for translation for the first time. It was a big thing. It really improved performance for the first time over the old methods. But it was still 25 blue. And training this model took about a month on 128 GPUs. It was absolutely not reproducible. Outside of Google, no one could do this because it was very tied to Google specific hardware. It was a special framework, but it worked. So RNNs started spreading into NLP. And as you learn in this course, the RNNs work by processing the input word by word and then decoding the output in another language like this. So RNNs are great, but as I told you, it's slow. It takes hundreds of GPUs. And also when the sentences get long, when you go beyond one sentence to paragraphs, the gradients don't propagate that well. Then in 2017 we had this model called Transformer, which does away with the recurrence in our net. So it's way more parallel, it can use way less computer, and it gets BLEU scores on the order. In the paper it was 28. Then if you train longer, you get it to 29 with some improvements. Now we actually got it to 30. So on the Bleus core metric, it's on par with human translators. That's what you learn in Course 4. The Transformer Model is deceptively simple because it has no recurrence. It has this attention matrix that attends everywhere. And it turns out it works not just for translation, it works for a large number of NLP tasks. You can.

Lucas, I'm sorry to interrupt. I just noticed your camera's off. Would you be willing to turn your camera on so folks can see as well?

Sure. Sorry, I didn't notice that turned off. So yes, that's the transformer model. And since it does better with gradients in long sequences, and to understand why it does better with long sequences, you really need to understand RNNs and then look at attention so that's why it's a really good idea to pay attention in course three, understand the RNNs with their features and problems and then go to course four. And with the attention mechanism you can generate a really long text. So this is the text generated from the OpenAI trans. The thing in green is the only thing you input. It's a prefix and then it continues the story and it really reads like you could have written the story, which five years ago was something in the field of nlp. You would barely believe that the model can be generating coherent stories. What are the frontiers? Well, we are going into longer and longer sequences. We are getting models that are more able to reason. We're getting models that can do few shot learning. There is now a GPT3 model that does few shot learning and there are more and more use cases for these models. So it's not just translation these days, it's a lot of NLP tasks generation. It's also language to programming. There are a lot of exciting developments still going on, but the basics you learn in these courses. So join them, learn about them. And these days, luckily you can train your models in either a Google collab or even just on your own computer. I find this very exciting that you can train a whole translation model in your own environment rather than on hundreds GPUs and we can really explain to you every detail of how it works. I hope you will like it too. Thank you, Ryan.

Thank you Lucas. That's fantastic. And super interesting how it seems like bigger and bigger models have been sort of path towards better and better outcomes, but then at the same time making those models faster and faster so that they can be not just big and slow and only runnable at Google, but big and fast. Super cool. I like how you emphasize too that understanding the basics of some of these building blocks like RNN's art is really important to getting to the big complicated models. I hope we can talk more about that in the panel discussion, but right now I'd like to turn it over to Andrew Ng, founder of DeepLearning AI, and get his perspective on NLP and where the field's going. Over to you, Andrew.

Thanks, Ryan. Hey everyone, it's nice to see you here online despite the circumstances of the world. And thank you for joining us. What I want to do is share with you some thoughts on the state of NLP today. I know that many of you are looking into breaking into nlp, hence the name of this session that we're holding today. But what I thought might be useful would be if I share some thoughts on where I think NLP is. So the rise of AI, the rise of the deep learning revolution, the rise of machine learning and AI is leading to more specialization into subfields. I feel like my background is a machine learning generalist. But as a few matures, there are now more and more specialized tools in the most important application sectors of AI. And this includes of course, nlp, with lots of specialized tools that grew up and are native to nlp, like some of the things that Lucas talked about, but some of which, like Transformers, which had its roots as nlp, but it's now breaking into other sectors as well. And then also of course, computer vision, speech, audio processing, a lot of exciting work on structured data that's not widely appreciated in some of the published literature, I think. And then other sectors, everything from medical imaging to robotics to earth sciences, all of these different sub disciplines are building up specialized tools to take advantage of the deep learning revolution. Just now, Lucas alluded to the history of the growth of deep learning and nlp. And I thought, I want to share with you some of the history that I saw for that because it has implications for, for some of the work that all of us do, maybe some of the work that you might end up doing today. So the rise of deep learning, we all know about that. What happened from where I was sitting, what I saw was that deep learning as an application discipline first had its biggest impact in speech recognition. I remember it was about 10 years ago, around 2010, the work of Deng Li and Geoff Hinton and a few others really showed that deep learning can make a big difference in speech recognition accuracy. What happened? And I was leading a Google Brain team around that time, reading a lot of these papers and implementing algorithms. But this first wave of the injection of deep learning to speech recognition resulted in dramatically improved speech recognition system accuracy and a wave of new applications, everything from the smart speakers to voice assistants to practical voice search that you use on your mobile phone maybe. And injection of deep learning to speech improved accuracy and led to a wave of applications. I think the second wave was computer vision, where again it's well told the story of 2012, ImageNet, AlexNet. And that in turn subsequently led to a slightly later wave, but equally big, or maybe even bigger, I don't know, wave of new applications in computer vision. And so you know everything from today, some of my teams are inspecting things in factories using computer vision, or there are teams driving big machines using computer vision, or computer vision is enabling driver assistance systems, maybe in a car that you drive or will soon drive, even if we aren't all the way to complete, you know, self driving cars in all the ways we hope to get to see someday. And of course face recognition with all the problems and opportunities that creates too. But that's a huge way of applications. And then I think nlp, as Lucas alluding just now, came a little bit later. And actually remember when deep learning first his speech recognition, the community had a lot of internal angst. We had spent all this time managing features. What are we doing? Will this work? And then the field of computer vision had the same controversy and angst and discussions. And then I remember when my friends in NLP were going through the same thing and deep learning guys were saying, look, got to use deep learning. But the people that spent a long time in other techniques, it was a journey of transition. But I think because NLP had the wave of deep learning had hit NLP later, this rise of increase in accuracy came a bit later in NLP than speech or computer vision. And so I think there is actually a very large set of untapped opportunities in nlp. As in today we have very good speech recognition applications with still lots of opportunities, with quite a lot of vision applications. But I think the set of untapped opportunities in NLP is just massive because the wave of improved accuracy has come relatively later and there's still so many applications that just opens up that no one has really, really thought of yet or invented. So to those of you thinking of jumping nlp, I think this is actually a very good time. It's not a bad time to do visual speech recognition either. But the space of untapped opportunities in NLP feels huge to me. And then of course, in the history of deep learning, there's still these other disciplines that are still jumping in and enjoying these improvements in performance, which are also leading to improvements in lots of applications. So today NLP has tremendous momentum behind it. Here's a market study. I tend to take these things with a grain of salt, but significant rises in the value of NLP applications are projected by disorganization. Tractica these numbers frankly seem low to me, but I find the trend useful is the confirmatory trend. And if you looked at the growth of ACL and a big NLP conference, number of submissions is rising rapidly. So the volume of activity very much seems up into the right and maybe even accelerating for nlp. And what's exciting about all this, one of the magical pieces of NLP is that today probably all of you use NLP Dozens of times a day without even thinking about it. The applications have worked so well, they've disappeared in the background. You don't think of them as NLP anymore. It's just something you use every day without thinking about it. Today most of us use web search probably every day we talk about search. Web search is the obvious one, but lots of websites have product search capabilities. Go to the website, maybe buying something, looking for a movie, looking to buy something for your home. Search is not just a web search engine. Essentially pervasive on a lot of apps and websites. Summarization again when you do a web search, that snippet of text is really valuable. Autocomplete again in search functionalities and how we type useful all over the place. Anti spam is without antispam our email system just won't function. Machine translation is actually getting really good. It is interesting that and we'll see. Hopefully the world becomes a place where anyone can talk to anyone else even if we don't understand the same language. But I find that very exciting. Smart speakers, smart assistants, in addition to speech recognition portion of it, there's an important piece to understand the words you actually said and translate that to action or command and and nascent chatbots. There was a wave of hype in chatbots. I think the hype around chatbots has died down. But there are actually working chatbots today that provide real value even though the hype isn't quite there. But some of these are today's NLP applications, many of which still have significant room for improvements or significant room for reinvention. And if you learn about nlp, I hope that some of these could be sectors. Maybe some of you could dive in to build cool stuff to move, to move your company or to move the world forward. One thing that excites me at least as much, maybe even more is all of the future applications that no one, none of us have even envisioned yet. I don't know what the future applications will be, but these are some that I personally find exciting. I'm just one person. Some of you, maybe you have better ideas than I do. I'm personally excited about the application of NLP to education specifically to alter grading, giving feedback to people at scale. I think there's a lot of room for improvement in email tools. I know email feels like it's mature, relatively consolidated industry, but with rise of modern NLP and I think Google smart reply when you hit a button and get a reply back, that's just one small piece. I think there's actually a lot of room where sometimes I ask my collaborators to help me deal with emails. Some of the things I ask my collaborators help me with I think could be automated. So a lot of room for improvement. Synthesizing academic papers There is research on this, but I'm actually excited about. There's so much literature in many disciplines like medical discipline. You know, no human can read in a small fraction of these papers. But I think there's actually a lot of I'm seeing nascent progress, I'm seeing weak signals that this could become a big thing in terms of spotting trends in statistical way from this giant body of literature to help us drive medical research or medical, maybe even medical decision making in a heuristic way sometimes. RPA robotic process automation has been taking off rapidly. Lots of great companies doing RPA and I think there's a potential for NLP to make RPA robotic process automation much more flexible and many more the is really I would love if a few years from now I'm giving another talk and maybe something that you come up with ends up what I could talk about and cite you in the some idea that none of us that I certainly have not yet thought of today. The rise of NLP's accuracy opens up tons of rooms both to improve today's applications, which are already important and pervasive, and also to envision new things that just were not possible before. And on the research front I want to highlight one which is just one other thing I'm excited about. And if you complete the NLP specialization, all the courses you'd be well positioned to jump into, try your hand at some of these cutting edge aspects. You may have heard a bunch of us, right Me, Yang Lecun, Jeff Hinton, also Yosha chat about the importance of unsupervised learning. Well, there's one aspect of unsupervised learning that I'm excited about that isn't like a 30 year off thing, but it's something that's working now. Well, I say about 10 plus years ago, I think it was actually like 12, 13 years ago I started getting excited about one form, one application of unsupervised learning that we call self taught learning. And here's the idea. You want to learn from a large amount of unlabeled data. The world has a lot more unlabeled data than labeled data, be it unlabeled images, unlabeled text. So learn from tons of unlabeled data. And this thought was motivated also by the observation that most of human learning is from unlabeled data. So some of you know I have an 18 month year old daughter called Nova and I love her, but I'm not going to point out one object to her every second of her life. So frankly I do my best to teach my daughter, but most of what any son or daughter, any infant learns is just wandering around by themselves with a little bit of supervision maybe, but. So we think most of human learning, human infant learning is unsupervised. So can we algorithms learn from tons of unlabeled data and then the self taught learning task is to take what you've learned from tons of unlabeled data, be it images or text or audio or whatever, and transfer that to different tasks. Usually a supervised task. We have only a minuscule or much smaller label data set. So we call this framework self taught learning. Self supervised learning, which I've talked about in other forums, has been exciting piece of progress in self taught learning. And I think the best example of this, the most exciting part in this is in nlp. So you heard Lucas talk just now and the thing that really enabled this type of learning to take off was the work that Lucas and his collaborators have done on the Transformers. One thing I've learned about Transformers work is that when you look carefully you find that there is more than meets the eye. That Transformers work enabled the rise of other great pieces of work like Bert by Google and the various flavors of GPT by OpenAI. I actually just read the GPT3 paper. Great work. And rather than using fine tuning instead giving a text prompt to GPT3 to have output additional output, I thought that was actually really really interesting and clever. But if you haven't seen some of the demos of GPT3 on the Internet, it's wor searching around and finding some cool demos. But this is an example of English language correction. This is actually an example drawn from the GPT3 paper. But I thought this was really exciting example of how this type of framework we learn from a massive amount of unlabeled data and then adapt it to some other task, maybe a supervised task, maybe something else in the case of GPT3 is giving a text prompt and having it do auto complete to do these things. I think this will open up tons of applications. I want to share just one last thought before I wrap up, which is just now. Lucas alluded to the early transformer networks, some of the machine translation models, the only stuff that you could do at Google back then. We've learned in tech that yesteryear's big giant supercomputer is tomorrow's cell phone wristwatch. And so all of these models, we'll see how long Moore's Law holds up. Maybe it will, maybe it won't. But I think all of these things. And by the way, actually, I remember early days when I was leaving Google Brain. I remember we were so proud of ourselves that we got 16,000 CPU calls to train a giant neural network. And at that time, only Google could have done that. Probably today, any of you can go online to one of the cloud services. And for about $3,000 was my back calculation, for about $3,000 worth of cloud credits, which isn't nothing. $3,000 is a lot of money. You could replicate what we were so proud of doing slightly less than a decade ago. So all of these things seem so incredible when we work hard on improving these algorithms and finding right applications, even if it seems niche. Just imagine one day where these models that we read about and are amazed by. I don't know if it'll ever run on your wristwatch, but. But this is how technology becomes more available. And right now I see a lot of weak signals in the field of NLP from really building off Lucas and NFT's transformative work. There's so many puns you can make with this stuff that I think will explode and become more capable and more widely available. And I hope that maybe you watching this from home, wherever you are, maybe by learning these tools, hopefully you can also participate in that revolution. Thank you.

Thanks, Andrew. I think that's. That's really cool what you say about the future of nlp. There's some things you can think about as exciting advances that might be just on the horizon. But perhaps the most exciting thing is the unknown, the things that nobody knows or nobody could predict. It's exciting to think about. So right now I would like to continue the conversation by welcoming our other panelists. So today we have online with us Kenneth Church, distinguished Scientist at Baidu usa, Marty Hurst, a professor in the School of Information and the Electrical Engineering Computer Science Department at UC Berkeley, and Eunice Mori, an instructor of AI at Stanford University. So before we start the panel discussion, I thought it'd be good to just get a little bit of an intro to each of you. So, Ken, if you could start us off. Ken Church from Baidu usa. Oh, I'm sorry. There you are.

Yeah. So you want an intro of.

Yeah, just an intro and a. Hello. Hello.

So I'm Ken Church. I work at Baidu in California. But right now I'm in New York. Any rate, I've been working in the field since the 70s, mostly in computational linguistics, but also speech and language and even things like data mining. Anyway, it's a real pleasure to be here.

All right, thanks Ken, we're really glad to have you. Marty Hurst, could you give a little intro to yourself and your background?

Hi everyone, my name is Marty Hurst. I'm a professor at UC Berkeley. Once, a long time ago, I was a grad student and worked with Ken Church. I visited him at Bell Labs and I'm actually an interdisciplinary scholar. So NLP is one of my fields. Another field is human computer interaction or hci, as well as information visualization and search. So I do focus on applications a lot of the time, although I can. I've been doing this since the late 80s, so I have done some foundational work at that time as well. And I really appreciate. I'm also very interested in accessible online education like Andrew, and so I applaud this kind of course where that you all are offering here.

Great, thanks very much for the intro, Marty. Eunice Mori, could you give a little intro to yourself?

Thanks, Ryan. I'm Eunice and I teach here at Stanford and also at DeepLearning AI, so you've probably already seen me at DeepLearning AI with Lucas in the specialization. And I'm also very interested in accessible education specifically for AI and hope to see you all soon.

I just say really grateful to Ken, Marty and Eunice in addition to Lucas for being with us. I just want to give a shout out to Ken, who is a huge help to the whole team in creating the NLP specialization in addition to being an insightful leading figure of nlp. So thank you Ken and Marty, I've known for a long time, had the privilege of learning from her, interacting with her way back when I was a student at UC Berkeley and Lucas always work with and know Lucas for a long time and Lucas and I actually co instruct one of Stanford's machine learning classes. So really glad to have all of you here and Lucas of course as well. Great.

Yeah, this group goes way back. So what I'd like to do now is for the next 30 minutes or so I'll be asking some questions of our panelists and while I'll be directing my question to an individual one of you, I'd like to encourage anyone else to feel free to jump in and offer your own input on the conversation. We'll just kind of keep it rolling that way. And then after that we'll be fielding some questions from the. From the submissions that we're getting online. So, Marty Hurst, I'd love to start off with you as a professor. I'd imagine that you have students coming to you a lot, asking what to focus on or what to get into or why to get into something like AI in general or nlp. So what do you tell them?

That's a great question. And we have both undergraduates and graduate students. And I also do talk to people that are looking at second careers, and I speak at panels as well. And that might be. I think a lot of your audience might fall into that group. So the first thing is, why are you considering this area? And I do find that people often do something because everybody else is, and that's often not a good reason. So really look carefully at what your motives are. Now, there's nothing wrong with dipping your toe in. And with these online courses, it's so wonderful that you can just get your feet wet, start to learn about something without a really big commitment and learn a bit about it and see if it's for you. But I think really assess what interests you, what you like, and then see if something about that field resonates with you. In terms of other parts of AI, they're not really my specialization, but I can speak to NLP and what drew me and what's drawn other students that I've worked with. So I'm really interested in language and, and I think that when I was in grad school, in my first years, people in nlp, there were a lot of people that were linguists, we were computational linguists. It was people interested in language or interested in psychology, interested in the mind, interested in philosophy. I see that there. And I was interested in sort of animation and how do we make a computer interact with language. I mean, that was just really fascinating. And I think today a lot of people are coming at it more from the machine learning angle and the mathematical angle, and that's a perfectly valid angle as well. And there's room in the field for both types of people or all types of people. But I think that it's, again, look at what your interests are and see which part of the field is right for you. But even if you are currently strong on one side of this one of these aspects, but not the other, don't let that deter you because you can learn the other aspect. So people that start with machine learning but don't know a lot about language can learn. I would say that language is a very deep field. It's Harder to learn than something like. Not harder, but it takes more time than something like information retrieval. Just language and linguistics is very deep, very complex, very rich, but you can still get a first pass at it. But it rewards years and decades of study. It never loses its interest, I would say. And the machine learning side, some people have that background, some don't. You can learn it. There's so much, there's so many great resources now, so different from when I was a student that you can, with study and practice, learn those skills if you don't have them. I'd say just the best thing about NLP is it's interdisciplinary. So it's not just the machine learning side. And there's applications that are fascinating, as Andrew so wonderfully outlined. And so if you're interested in applications, in changing the world and making a difference between for good, then there's a lot here for you. And if you're a skeptic too, I mean, there's a lot to question about AI, and I think there's a lot that you can contribute from that angle in this field. It helps to be well informed about the technology if you want to look at it from a skeptical perspective.

Yeah, that's a really interesting point. That the field would definitely benefit from more skeptics and more folks challenging ideas. It's also interesting what you said about NLP or language. I know you worked on computational linguistics, but there's. I think sometimes we make the mistake of thinking about NLP as just another application of machine learning, but it's really language meets machine learning and language itself is a whole world unto itself. So that's, that's an interesting thing to consider. I'd like to ask you a question. Ken Church, you've been in the field a long time. You've worked on a lot of different things. You've seen the academic perspectives, the industry perspectives. First, if you could just kind of paint a picture of where the field was when you got started in it and where it's come to today. I know that's a broad scope, but if you could give us some historical perspective, that'd be great.

Sure, I'd love to. So actually, let me go a little broader than you suggested. Was at Harvard when Skinner was there, and in those days, empiricism was all the rage, not only in psychology, but Shannon was sort of at his peak then. Information theory. Firth quoted this very famous quote, you shall know a word by the company it keeps. Harris, who was Chomsky's professor, pushed the distribution hypothesis. The idea that you would understand what's going on by distributional statistics all feels very familiar again today. Then 20 years later, I was at MIT when Chomsky and Minsky were there and they were pushing a different position. That was called rationalism. It was anything but the empiricism of the the 50s and then my generation. So what I want to say is that Chomsky and Minsky rebelled against their.

Teachers.

Against our teachers. Chomsky and Minsky, we revived empiricism. And it's hard to remember that before the 1990 or so, it was almost impossible to get a paper into a top conference that use statistics at all. And then 20 years later came along deep nets. So what I want to say is that every 20 years there's an oscillation. It goes back and forth. And I think the basic common theme is like that cliche about grandparents and grandchildren. They have a common enemy that every generation needs to rebel against its teachers in order to make its mark. And I wrote a paper called A Pendulum Swung Too Far, which sort of goes over this whole history anyway, in retrospect, maybe that's sort of unfortunate metaphor that it's not really quite as simple as that. Chomsky was a strong personality. Minsky's actually a sweetheart deep down, although he never really warmed. I want to say that while, you know, we never agree with all our teachers on everything, we don't agree with everyone on everything. And it's perfectly fine to disagree. What I want to say is that a lot of times fads come, fads go, but a lot of these things have always been around in good times and bad times. Hinton and Lecun did what they did when it was fashionable and when it wasn't. And similarly Saltin, who's sort of the guy behind vector space, which is really like Word2VEC, and it's really the foundation between behind much of what we do. That stuff was very unpopular through most of his career. So I want to say that we mostly have been talking about methods, empiricism, statistics and so on. I think Marty did a pretty good job of talking about some. That's important too.

It was very important.

We didn't always have with data. Back in the 80s, it was very, very hard to come up with data. Nowadays it's a lot easier. You can hope to get massive numbers of books and Wikipedia and all sorts of things. You can download these wonderful data sets. It wasn't so easy. But even so, we tend to underestimate the importance of things like balance Corpora A data set should be a sample of something we want to think about what the population is we're trying to sample. Not all data is interchangeable and so on. The last thing I really want to highlight is the commercial impact. I no longer have to explain to people what I work on. There was a time when I had to explain what speech recognition is, what speech synthesis is. You've all experienced this now. There was a time when there really weren't any commercial successes. You probably heard of the company Symantec, but you probably don't know why it's called Symantec in our area anyway. Nowadays there are lots of Fortune 500 companies in this area. I no longer really have to. We don't have to talk about just supply. There is tons of demand for what we do. That's all new. That wasn't like that in the 50s, certainly not even in the 60s and 70s or 80s or 90s. Anyway. Thank you.

That's fantastic, that perspective. Yeah. Working on things when they're not in demand yet. But now for sure, there's no question about whether this stuff is relevant or whether it's going to change the world. That's very interesting. Lucas, I'd like to ask you a question. We talked a little bit or you talked about Transformers? A little bit. And Andrew touched on it a bit in his talk as well a few years back. The paper you wrote or co authored with folks called attention is all you need and we've already had some good puns on this, but this paper's gotten a lot of attention, if you will. And the subsequent things that have come out based on Transformers have been really exciting, I think for the audience that we have gathered today. I think many of whom might be familiar with the sort of math and basic architectures around neural networks. How would you explain what is the transformer and what it was designed to achieve?

Well, I think it's good to bring back to what Ken said. Everything comes in waves and everything builds on top of the things that were before the. You need to understand the context before transformer, which is where everything was. At least in the deep Learning part of NLP. RNNs were doing everything. They were the big first wave of deep learning nlp and they have this super basic idea that you should do things in recurrent step by step process, which is so intuitive, but it's really slow on. On modern hardware because the accelerators are built for parallel processing. So the idea of attention comes from another very old technique in nlp, which is alignment. When you try to translate sentences. You have an English one, you have a French one. You try to align which words correspond to which words. So if you put this alignment idea into a neural network, what you get is the attention mechanism. It is a soft version, a differentiable version of aligning words with words that come before. And self attention is aligning the same text such that you align with words that appeared earlier. And this idea works surprisingly well in terms of the results you get, but it also works surprisingly fast if you implement it well on modern deep learning accelerators. So it's a mixture of making the implementation really easy to use. And I think that that was part of why it became so successful, because you can just run it and it runs fast enough. It's always a big difference whether you need to wait for your result one day or one month. But the other thing is alignment is a really good idea. This has been known in NLP for, for a long, long time. If you want to translate a whole paragraph, it's much easier when you focus on translating this part and only look at this one part at a time. That's what every human being does. And you give this prior to a neural network, and it works really well. So that's where attention came from. But I would want to say that as everything goes in waves, it's a very good idea to use RNNs because there are modern papers on sparse transformers that kind of start showing that this attention is more like RNNs than people think. And actually adding some recurrent networks into it makes it even better. So it wouldn't be surprising if people go back to recurrent networks in some way of fashion. Again, that's why it's good to learn about more than just the most recent technique. It's good to know things that came before and learn all about it.

That's really interesting that in some sense with the development, pushing the boundaries of Transformers, that you end up kind of somewhat circling back to where you came from. Yeah, it again highlights that the building blocks are important or that where it came from is important. Eunice, I'd like to ask you a question. You're the sort of earliest career panelist here, and I think it must be an interesting perspective to have been studying all this stuff or sort of developing your own mastery while the field was advancing so fast or while it was sort of making such a big public splash. So maybe you could give some perspective on, on what your journey's been like and also any recommendations you'd have for people who are Wanting to potentially start on that journey themselves.

Yeah. Thanks, Ryan. So my journey has been a little bit different. So for me, education means a lot to me and especially I grew up in Morocco, so we do not have access to the latest technologies and state of the arts and as many resources as we have here at Stanford. So for me, I really wanted to be part of the mission that will help take the latest technologies and state of the art and make it accessible to everyone around the world. So that's how I started. And I came here around in 2014 where I started with the machine learning course on Coursera. And I started by writing like lecture notes and trying to make it as accessible as possible to everyone. And eventually I started building up and studied math, computer science, statistics, and just started slowly mastering building assignments and quizzes and so forth. So that was pretty much the journey. Then we built the deep learning specialization and we started teaching on Stanford campus. So it's been interesting because every time you, you finish building an assignment, then there's new technology that goes out and then you have to read another paper and then try to transform it into another assignment and it just keeps going on and on. So it's been definitely exciting to be like, you know, as soon as you're done with a new assignment, you, you go ahead and try to bring in new states of the arts.

So, yeah, I can definitely say from my own perspective that working on these courses with you and Lucas has been really interesting in the fact that the field has developed significantly while building the courses. So that feels like it's been an exciting, an exciting path. Andrew, I'd like to turn to you with another question. So as you mentioned, you worked on a wide range of AI applications. I think that one of the issues that you've also thought about is that the fact that AI often seems like a black box and people struggle with explainability, or at least there's skepticism around the, the sort of black box nature of AI algorithms and particularly the bias that might come out of algorithms when the output's not fully understood. So in the field of nlp, what thoughts do you have on how models can, or how we can think about bias and, and what the. I don't know what the big considerations are.

Don't know this tough one. You know, I think that I've seen a few pieces of very good work. One of the most influential papers of recent memory was the one out of Microsoft showing the bias in learned word embeddings where if you learn from text off the Internet learns the really Horrible analogy. That was it man as the software engineer, or man is the computer programmer as women into the horrible output homemaker or something like that. And to the Microsoft Team's credit, they proposed a solution to zero out this form of bias so that you end up with the more appropriate conclusion, which is man is to compute program up as woman is to compute the program up, which seems like a much more appropriate way of mapping analogies. I think explainability and understandability is one of those things that speaks to the importance of the field. Once upon a time, the field of AI, frankly the field of computer science had much less of an impact. And it was just a few people writing codeware, writing software, doing cool things, trying to build things that work. But today, the work of the entire discipline, computing AI, nlp, has such an outsized impact that we owe it to society really to think through the real impact of our work and then to do our best to move things in a positive direction. And I think there's one. When Ken and Lucas were speaking, there was one thing that occurred to me that I want to comment on, which is Ken mentioned, Lucas mentioned also technology coming in waves. And Ken used the term fads in nlp, which I think is completely accurate. One lesson I've learned thinking a lot about technology is I think it's important to time the technology appropriately. Who invented the helicopter, that thing that flies around it Turns out Leonardo da Vinci had early drawings of what looks a lot like a helicopter. There were some problems of Leonardo da Vinci's designs. He did not get it right. But if you search online, you find Leonardo da Vinci was trying to invent a helicopter. Totally did not work. Without the internal combustion engine, the human body could not generate enough power for its contraption to live on in the air. So it failed. How about smartphones? Well, the Apple Newton. Search online if you want. The Apple Newton was a terrible idea. Technology wasn't ready. LCD screens weren't ready, batteries weren't ready, the touch interface wasn't ready. So the Apple Newton, a couple decades later, Steve Jobs launched the iPhone and that transformed the world of mobile computing. So I think a lot of the fads that can accurately alluded to is about getting the timing right. In my opinion, and this may be controversial, 30 years ago, deep learning was a terrible idea. Computers weren't fast enough, the data wasn't there, it just didn't work. Now I'm not ungrateful to people, they're working on deep learning 30 years ago. I think they laid the foundation. And you can also Argue maybe. I actually have no idea if Leonardo da Vinci's early drawings of helicopters inspired what actually happened about 50 years ago or something. Actually, maybe more than 50 years ago when it became practical. And even though Apple Newton failed, you could argue maybe laid the foundation for the modern smartphone. So I'm not dismissing the importance of the early foundational work, But I think 30 years ago the timing was wrong. And the right timing to jump into deep learning was maybe 15 years ago when finally the compute became powerful enough and the data became there. And then deep learning kind of finally overtook Support Vector Machines or whatever we're doing a little bit before then. So really, Leonardo da Vinci deserves tons of credit for envisioning the helicopter, even though his attempts to do it failed. But I think as we go through these things, I think the timing of nlp, it feels more solid now that the computer's there, the data is there, we're getting early nascent results. And one of the ways to have an impact on the world, not the only one, we need people work on all sorts of things, including things like 50 years out or including things that are 100 years out, that's fine. We need people working on all sorts of things. But if you look at the weak signals, I think one of the best ways to make a big contribution to society is if you can spot the weak signals about things that without hope, can take off in a better way or faster way over the next several years. And to me, NLP pattern matches very well. You can see a lot of the weak signals that suggest that if you work on it, you can make a big difference. Whereas arguably, I think maybe Leonardo da Vinci did have a big impact on the. The rise of Holocaust is as you don't know. And Certainly the Pioneers 30 years ago had a big impact on the world today. But given that we all have limited lives and limited amounts of time, I want to do the work where I had kind of the biggest impact. And so I tried to think through if I do this now versus don't do this now, what enables me to hopefully help others the most. And I do think NLP matches up to that. Yeah, and I don't know, and I know, actually, Marty, I know you thought a lot about timing too. I don't know if Ryan wants us to jump in chaotically.

Yeah, I do, actually. I think that I was going to follow up there. But Marty, I would love to get your perspective on timing because, yeah, I think I've also. I guess there's one way to think about it, like if you want to time something right, you should be ahead of the wave. And so, like, if it's a big deal now, maybe it's too late. You should move on to the next thing. But it sounds like for NLP at least, even though it seems like a really big deal right now, it's perhaps just the beginning or something like that. But, Marty, I'd love to get your perspective.

Well, first, I loved Andrew's analogy to DaVinci and the helicopter and neural nets not being ready. When I was a grad student, people were tinkering around with it and hoping to make it work. Actually, a lot of people I know got interested in AI, got sucked into neural nets, and it kind of was. It was good, they were productive, but it wasn't really going anywhere then. So just being the wrong timing and linking that to something seeming like a fad as opposed to, oh, it's having its moment because it's possible. That said, there's a lot of things that become fads that are just popular because there's something that interests people and even though they're clearly not working. So I think that does happen as well. And I can think of examples that I won't name, but in terms of sort of timing, in terms of choosing research projects, and I know this is an audience, I think maybe largely of practitioners, so I don't know how much it applies to people in the audience here, but for those who are thinking about research, my strategy, or I guess I couldn't call it a strategy, my bent was always to look at an area where there weren't a lot of people working or look at a problem where people weren't that interested in it. Like I was looking at Search before the Web. So it was. To say it was unpopular is an understatement. It was very much a backwater. But I was really interested in the question. I just didn't like library catalogs. I wanted to fix it since I was a kid, and so that's why I was interested in that topic. And I had ideas about it since I was a kid. And now they are what you see on websites when you shop now. So that's very satisfying for me. So a lot of times there's problem that you want fixed and nobody else is thinking about it. So maybe you can fix it or maybe a few people are thinking about it, but you have a slightly better way of doing it. So for me, that's what I started with. It's what are you interested in? Not as what is everybody else interested in if you really want to have satisfaction in an intellectual pursuit, I think that's the most important thing to think about. And then if you want to pursue something that other people aren't doing, be prepared for a lot of rejection. Be prepared for your papers to be rejected. I have a lot of well cited papers. Most of them were rejected the first time I submitted. I often had to send them to a smaller venue or something, and then they were kind of cited. Strangely, a few of them were accepted the first time. But that happens to almost everyone who has a new big idea. Their papers are rejected the first time. If it's a good idea, just keep working on it. So, yeah, everyone has their own way of working. Some people are really good at taking ideas that are established, but making them deeper, making them more sound. And that's a great way to advance work as well. It's just so important to understand what are your particular interests and strengths and do that and not what everyone else is doing for that sake.

Yeah, absolutely. Ken, I wanted to get your perspective as well on the timing of things. I know you've probably seen a lot of different things timed well or timed poorly, and what would you say?

So I think of timing a little bit like a stock market. You want to buy low and sell high and not vice versa. And a tendency is to do just the wrong thing. That is, it's sort of like a kid's soccer game where everybody goes to the ball. It's much better to play your position and to get a good assist. And in my own case, you know, I was very early on for the revival of empiricism in the 90s. My first empirical paper was in the late 80s. People have been looking at it. In those days, maybe there were hardly any papers in empirical methods. A decade later, there were hardly any papers in anything but empiricism. So it's good to get started in the 90s, and by a decade later it was time to do something else. Deep nets came in a pretty good time. I would worry about jumping in when everybody else is jumping in and doing what everyone else is doing. You want to get into the next new thing. And I don't know what that is, but, you know, I think we could make some suggestions. I don't think it's a good idea to be doing what everyone else is doing. And Marty, I think, put that very well. Thank you.

Yeah, I think the audience would certainly welcome suggestions, if anyone has them. I know that it's hard to predict what the. What the thing will be but.

So maybe I can jump in with one comment on that. I feel like it might feel with the Internet, like everyone is doing deep learning in nlp. I feel like it's actually the opposite. I think this is a field that has barely scratched the surface. Of course, nowadays everything happens online. It feels like if you submerge into a field, it feels like there is a lot of movement because the world of NLP is much bigger than it was decades ago. But I feel we are barely scratching the surface in what these deep models can actually do to language. And this is, as Marty says, language is an extremely deep field. Where we are, yes, we can generate you a story. And the deep. It reads okay, but does it really have meaning? Does it really is. You know, can we really steer this to generate things that are true? Can we verify? Can it tell us, you know, we talked about biases. You can actually ask the language model is this offensive? And in a lot of cases it will tell you whether it is or not. But it will make some mistakes that are very non human. Like, can we understand why? I feel like with the GPT3, which was only released weeks ago, it's the first model that shows that you can do learning without gradient descent. You can just put things into the model as input and it works as if it was training on them. So this is very new and it's very hard to test because you need large models. So I feel like in half a year, in a year, there will be models that, that you'll be able to start doing this with hands on. So that's why I feel like it is a great time to start learning because it's only beginning a lot of these things that actually go into language. I feel like we've gotten the technology, we've gotten the connection to deep learning, but actually going into the depths of language is still ahead of us very much. And this will bring up. I think maybe Marty can say more about that.

Yeah, I think you're making a great point, Lucas. And I don't want to imply that people should stop doing deep learning. I think that's a fundamental tool now. So one works with the fundamental tools of the field and moves forward. And I didn't want to imply otherwise in case I was. I think that the key thing though is like you just raised GPT. So I think a lot of people are having reaction to language modeling as being the way. Now we're getting a little technical here, but being the way to solve all the problems. And so people are looking for alternatives to that since it's clearly not going to solve, I think. I think it's not going to solve a lot of the hard problems. It doesn't model a lot of things about language and communication that are really important and by the way, for an area for people to think about going into that. Andrew mentioned chatbots, for example, which are sort of a type of communication. We really don't know how to do communication well with language or interaction. That is really unsolved. And it's an intersection of a lot of interesting fields. We really don't. Our machines do not read, despite that word. There's no understanding in these systems, I don't think. And so we are. We're doing really well on applications that we did terribly on 10 years ago. I used to open one of my classes with scenes from 2001 A Space Odyssey and have the students say what's possible and what's not. I don't even do that anymore because they're almost all possible now, except for the very end of the movie. But nonetheless, it's somewhat of a mirage or an illusion in terms of what's going on with nlp. And there's. So instead of just working on. What I mean is, don't just work on language modeling, work on these other more difficult problems, which I think is exactly what Lucas is saying.

Absolutely. Yeah. That's interesting perspective. Ken, did you have something you wanted to add to that?

Yeah. So I certainly agree there's tons of interesting open problems in language and deep learning, and I don't want to discourage anyone from that. What I'm more worried about is that there are a lot of papers these days on sort of mindless benchmarks. And I've got an opinion piece coming out soon called Benchmarks and Goals, where I'm saying we should focus on the goals. What are we trying to address here? And I worry that some of the benchmarks take on a life of their own, that they were posed for no particularly good reason. They might have once made sense, but they certainly don't make sense now. And suddenly you see an enormous number of papers showing incremental progress on something that's not worth doing. And that worries me. Okay, so I think. Here there are a lot of things worth doing, but that's not one of them.

Absolutely. Yeah. I think that it can seem like that sometimes that there's just endless plots showing the number moving slowly higher and higher.

Well, on tasks that are not worth doing.

Can I say on one hand. Can I be. On the other hand, I think to A lot of people, I think it's okay. So there are a lot of online programming competitions or some of the machine learning modeling competitions. I think at some point they are pointless, but I think to a lot of people it's an important learning experience. So when I was learning, actually, I am still learning AI myself, but in my phases of learning journey, I do a lot of pointless things, write a lot of code with no purpose other than I want to try this thing out and see what happens. And I think it's actually okay. I was talking to someone in a large travel company that had built a chatbot and their executives were saying, hey, this is useless. How is this helping this big travel, this very large travel company? How is this helping the company make money? And I think they said, no, it doesn't help the company make money at all. But I think that's fine because to this team in this very large company that I think you probably have heard of, I'm not going to say the name, it was an important part of the learning journey of the AI team to try these things out, do a little demo and the go. Now, if they stay at that level for their whole life, then that's not great. But as a stepping stone to then do something bigger and more cyclical, I think so, I think really, to anyone, go ahead, download some silly thing. Some silly thing. Download tweets and classify things with it. It's totally useless. If that's all you do your whole life, maybe that's not so great.