Deep Learning for Natural Language Processing (Richard Socher, Salesforce)

Thank you everybody and thanks for coming back very soon after lunch. I'll try to make it entertaining to avoid some post food coma. So I actually have a lot to owe to being here to Andrew and Chris and my PhD at Stanford here. It's always fun to be back. I figured there's going to be a broad range of capabilities in the room, so I'm sorry, I will probably bore some of you for the first 2/3 of the talk as I'll go over the basics of what's nlp, natural language processing, what's deep learning, and what's really at the intersection of the two. And then the last third I'll talk a little bit about some exciting new research that's happening right now. So let's get started with what is natural language processing. It's really a field at the intersection of computer science, AI and linguistics. And you could define a lot of goals and a lot of these statements here we could really talk and philosophize a lot about, but I'll move through them pretty quickly. For me, the goal of natural language processing is for computers to process or scare quotes, understand natural language in order to perform tasks that are actually useful for people, such as question answering. The caveat here is that really fully understanding and representing the meaning of language, or even defining it, is quite an elusive goal. So whenever I say, the model understands. I'm sorry, I shouldn't say that. Really, these models don't understand in the sense that we understand language anything. So whenever somebody says they can read or represent the full meaning in its entire glory, it's usually not quite true. Really. Perfect language understanding is in some sense AI complete in the sense that you need to understand all of visual inputs and thought and a lot of other complex things. So a little more concretely, as we try to tackle this overall problem of understanding language, what are sort of the different levels that we often look at? It often and for many people, starts at speech. And then once you have speech, you might say, all right, now I know what phonemes are, smaller parts of words. I understand how words form. That's morphology or morphological analysis. Once I know what the meaning of words are, I might try to understand how they're put together in grammatical ways such that the sentences are understandable or at least grammatically correct to a lot of speakers of the language. Once we go and we understand the structure, we actually want to get to the meaning. And that's really where I think most of the interesting, most of my interest lies in semantic interpretation, actually trying to get to the meaning of in some useful capacity. And then after that we might say, well, if we understand now the meaning of the whole sentence, how do we actually interact? What's the discourse? How do we have spoken dialogue system, things like that. Where deep learning has really improved the state of the art significantly is really in speech recognition and syntax and semantics. And the interesting thing is that we're kind of actually skipping some of these levels. Deep learning doesn't require often morphological analysis to create create very useful systems, and in some cases actually skips syntactic analysis entirely as well. It doesn't have to know about the grammar, it doesn't have to be taught about what noun phrases are or prepositional phrases. It can actually get straight to some semantically useful task right away. And that's going to be one of the advantages, that we don't have to actually be as inspired by linguistics as traditional natural language processing had to be. So why is NLP hard? Well, there's a lot of complexity in representing and learning and especially using linguistic, situational world and visual knowledge. Really all of these are connected when it gets to the meaning of language. To really understand what red means, can you do that without visual understanding? For instance, if you have, for instance, this sentence here, Jane hit June and then she fell or and then she ran, depending on which verb comes after she, the definition, the meaning of she actually changes. And this is one subtask you might look at, so called nephro resolution or coreference resolution in general, where you try to understand who does she actually refer to. And it really depends on the meaning. Again, somewhat scare quotes here of the verb that follows this pronoun. Similarly, there's a lot of ambiguity. So here we have a very simple sentence for words, I made her duck. Now, that simple sentence can actually have at least four different meanings, if you can think about it for a little bit, right? You made her a duck that she loves for Christmas as her dinner. You made her duck like me just now. And so on. There are actually four different meanings. And to know which one requires in some sense situational awareness or knowledge to really disambiguate what is meant here. So that's sort of the high level of nlp. Now, where does it actually become useful in terms of applications? Well, they actually range from very simple things that we kind of assume are given now, we use them all the time every day, to more and more complex and then also more in the realm of research. The simple ones are things like spell checking or keyword search and finding synonyms. And if thesaurus, then the medium sort of difficulty ones are to extract information from websites trying to extract sort of product prices or dates and locations, people or company names, so called named entity recognition, you can go a little bit above that and try to classify sort of reading levels for school text, for instance, or do sentiment analysis. That can be helpful if you have a lot of customer emails that come in and you want to prioritize highly the ones of customers who are really, really annoyed with you right now. And then the really hard ones, and I think in some sense the most interesting ones are machine translation. Trying to actually be able to translate between all the different languages in the world. Question answering. Clearly something that is a very exciting and useful piece of technology, especially over very, very large complex domains can be used for automated email replies. I know pretty much everybody here would love to have some simple automated email replies system and then spoken dialogue systems. Bots are very hip right now. These are all sort of complex things that are still in the realm of research to do them really well. We're making huge progress, especially with deep learning on these three, but they're still nowhere near human accuracy. So let's look at the representations I mentioned. You know, we have morphology and words and syntax and semantics and so on. We can look at one example, namely machine translation, and look at how did people try to solve this problem of machine translation? Well, it turns out they actually tried all these different levels with varying degrees of success. You can try to have a direct translation of words to other words. The problem is that is often a very tricky mapping. The meaning of one word in English might have three different words in German and vice versa. You can have three of the same words in English, meaning all the single same word in German, for instance. So then people said, well, let's try to maybe do syntactic transfer where we have whole phrases like to kick the bucket just means stauben in German. Okay, not a fun example. And then semantic transfer might be, well, let's try to find a logical representation of the whole sentence, the actual meaning in some human understandable form, and then try to just find another surface representation of that. Now of course that will also get rid of a lot of the subtleties of language. And so there are tricky problems in all these kinds of representations. Now the question is, what does deep learning do? You've already saw at least two methods, standard neural networks before and convolutional neural networks for vision. And in some sense there's going to be a huge similarity here to these methods because Just like images that are essentially a long list of numbers, a vector and standard neural networks where the hidden state is also just a vector or a list of numbers that is also going to be the main representation that we will use throughout for characters, for words, for short phrases, for sentences, and in some cases for entire documents. They will all be vectors. And with that, we are sort of finishing up the whirlwind of what's nlp. Of course, you could give an entire lecture on almost every single slide I just gave. So we're very, very high level, but we'll continue at that speed to try to squeeze this complex deep learning for NLP subject area into an hour and a half. I think there are two most, two of the most important basic Lego blocks that you nowadays want to know in order to be able to sort of creatively play around with more complex models. And those are going to be word vectors and sequence models, namely recurrent neural networks. And I kind of split this into words, sentences and multiple sentences. But, but really you could use recurrent neural networks for shorter phrases as well as multiple sentences. But in many cases we'll see that they have some limitations as you move to longer and longer sequences and just use the default neural network sequence models. All right, so let's start with words and maybe one last blast from the past here. To represent the meaning of words, we actually used to use taxonomies like wordnet, that kind of defines each word in relationship to lots of other ones. So you can for instance define hypernyms and is a relationships. You might say the word panda, for instance, in its first meaning as a noun, basically goes through this complex dag, this directed acyclic graph. Most of it is roughly just a tree. And in the end, like everything, it is an entity, but it's actually a physical entity, a type of object. It's a whole object, it's a living thing, it's an organism, animal, and so on. So you basically can define a word like this. And another way, at each node of this tree you actually have so called synsets or synonym sets. Here's an example for the synonym set of the word good. Good can have a lot of different meanings, can actually be both an adjective as well as an adverb as well as a noun. Now what are the problems with this kind of discrete representation? Well, they can be great as a resource. If you're a human, you want to find synonyms, but they're never going to be quite sufficient to capture all the nuances that we have in language. So, for instance, the synonyms here for good were adapt, expert, practice, proficient, and skillful. But of course, you would use these words in slightly different contexts. You would not use the word expert in exactly all the same contexts as you would use the meaning of good or the word good. Likewise, it will be missing a lot of new words. Language is this interesting living organism. We change it all the time. You might have some kids, they say yolo, and all of a sudden you need to update your dictionary. Likewise, maybe in Silicon Valley you might see ninja a lot. And now you need to update your dictionary again. And that is basically going to be a Sisyphus job. Nobody will ever be able to really capture all the meanings. And this living, breathing organism that language is. So it's also very subjective. Some people might think ninja should just be deleted from the dictionary and they don't want to include it. I just think nifty or badass is kind of a silly word and should not be included in a proper dictionary. But it's being used in real language and so on. It requires human labor. As soon as you change your domain, you have to ask people to update it. And it's also hard to compute accurate word similarities. Some of these words are subtly different, and it's really a continuum in which we can measure their similarities. So instead, what we're going to use and what is also the first step for deep learning, we'll actually realize it's not quite deep learning in many cases, but it is sort of a first step to use deep learning in NLP is we will use distributional similarities. So what does that mean? Basically, the idea is that we'll use the neighbors of a word to represent that word itself. It's a pretty old concept. And here's an example, for instance, for the word banking, we might actually represent banking in terms of all these other words that are around it. So let's do a very simple example where we look at a window around each word. And so here the window length. Let's just for simplicity, say it's one. We represent each word only with the words one to the left and one to the right of it. We'll just use the symmetric context around each word. And here's a simple example, corpus. So if the three sentences in my corpus, of course we would always want to use corpora with billions of words instead of just a couple. But just to give you an idea of what's being captured in these word vectors is I like deep learning, I like nlp, and I enjoy flying. And now this is a very simple so called co occurrent statistic. You'll just simply see here. I for instance appears twice in its window size of one. Here the word like is in its window and its context and the word enjoy is once in its context. And for like you have twice to its left I and one's deep and one's nlp. It turns out if you just take those vectors now this could be a vector representation. Just each row could be a vector representation for words. Unfortunately, as soon as your vocabulary increases, that vector dimensionality would change and hence you'd have to retrain your whole model. It's also very sparse and really it's going to be somewhat noisy if you use that vector. Now another better thing to do might be to run SVD or something simple like PCA dimensionality reduction on such a co occurrence matrix. And that actually gives you a reasonable first approximation to word vectors. Very old method works reasonably well. Now what works even better than simple PCA is actually a model introduced by Tomasz Michalov in 2013 called Word2Vec. So instead of capturing co occurrence counts directly from out of a matrix like that, you'll actually go through each window in a large corpus and try to predict a word that's in the center of each window and use that to predict the words around it. That way you can very quickly train. You can train almost online, though few people do this and add words to your vocabulary very quickly in this streaming fashion. So now let's look a little bit at this model word 2vec, because it's first a very simple NLP model and two sort of is very instructive. We won't go into too many details, but at least look at a couple of equations. So again, main goal is to predict the surroundings.