DeepLearning.AI NLP Learner Community Event ft.Valeri Voev
Valeri Voev, a former econometrics researcher, describes an NLP problem at LEGO: identifying products from messy text sources like retailer reports and social media. Simple regex catches 70% of cases, but the rest requires handling multiple languages, misspellings, and a thousand evolving product classes with few labeled examples.
Chapters
Hello everybody. My name is Valeri Boj. Welcome to this event organized by Deep Learning AI for the NLP learner community. I'm very excited to be here and I'd like to thank all of you for joining this event. I'm hoping that I can make you time worthwhile spending here some 30, 40 minutes. I also like to thank the organizers from Deep Learning AI, Alice Lin in particular for planning this event. It went all very smoothly. My plan for today is to Talk for about 15 minutes and then we'll go to the Slack channel and see what kind of questions you have. So put questions, upvote questions and then we'll see kind of what I can answer in terms of time and also in terms of my abilities and whatever I cannot answer due to time. In particular, I'll try to answer later in the channel. Okay, I will share my presentation now. So to briefly introduce myself, I have a PhD in econometrics with specialization in time series. That was some time ago. I got my PhD in 2008. The title of my dissertation was Three Essays on Estimation and Dynamic Modeling of Multivariate Market Risks using High Frequency Financial Data. It's a mouthful, but so what it was about is that around this time, in the middle of the 2000s, 2005, 6, it was very hot. This concept of high frequency financial data, that's when various stock exchanges around the world were providing data on all the trades and all the quote updates. So I was working mainly with data from the New York Stock Exchange where you could actually purchase this data. And back then, one day of trading on the New York Stock Exchange was one CD worth of data. So we would get These boxes of CDs sent over from the New York Stock Exchange and we would do like time series analysis on this data with my colleagues at the university at that time. So I remember that we actually had to copy all the disks to a network drive storage because otherwise to get a time series, let's say of 30 days of trading, you would need to swap, you would need to swap 30 CDs. So that wasn't fun. So we made our own big data center by buying a cabinet, drilling a few holes, putting couple of ventilators and there you go. So we had our data there and we could analyze it much more easily. But those were the days where you were getting data via CDs over the mail. So then later when I graduated, I had a position as an assistant professor at Orcs University in Denmark, where I am also based right now. And I should probably say that we are streaming this from my home right now. So I have children in the house. So in case there's some interruption, you know, be patient, but I hope it won't happen. So in 2008 I was at the Econ department at the center for Research and Econometric Analysis of Time Series. So you kind of get the picture. I was mostly working in time series things like autoregressive models, in particular volatility models, if some of you have heard things, things like general autoregressive, conditional heteroskeleticity or garage models. And then there you say more recent cousins that are linked to high frequency data, something called realized volatility and realized correlation. After three years as an assistant professor, I got this interesting opportunity to to join one of the leading wind turbine manufacturers. It was called Siemens Wind Power at the time. Now it's called Siemens Comessa Renewable Power where I was heading a small team where we were doing risk management and data analytics, mostly involved in the risk assessment of long term service programs which are similar to an insurance contract where you insure the machines, basically the wind turbines, and that will be maybe a 10, 15 year contract. So we would go and try to analyze what kind of risks are associated. Cost in particular is associated with signing an insurance, with selling an insurance contract of that type. And in the meantime I changed over a couple of years ago, almost four years ago now, to the LEGO group where I'm currently a lead data scientist at one of the data science subunits called Shopper and Consumer Engagement. And I'm going to show you a small use case from my current job. It's nothing too advanced, but you know, I think not everything that you will be doing for those of you that are not yet in a job like this, it's not going to be all that fancy as you could, as you could do with all these techniques that you are learning in this specialization. So the idea is the business problem is very generally identify a specific LEGO product from some text and where can this text come from? Could be multiple sources. There could be customer call transcripts with the consent of the caller, where we transcribe the calls. It could be our retail partners providing us with information about the sales of our products for Lego. It's much more important that the Lego sets get in the hands of children. Which means that if we sell to a retailer like a large retailer in the US could be Target, could be Walmart here in Denmark, there's other big retailers we sell to the retailer doesn't mean that the product has reached the final consumer, which is the children and of course adult fans as well. We have a very lively adult fan community. Nevertheless, we would like to know how many of our products are actually getting in children's hands. And for that reason we get some information from our retailers which of our products are actually selling. And sometimes especially smaller retailers in Europe is much more small retailers. They don't necessarily provide a product number or something like this. It could be just some text. So we try to infer what the product is. It could be some engagement from our fan communities on various social media channels where we try to identify which products are people talking about and with what kind of what are the sentiments around these products. So I'm not going to be talking too much about the sentiment now. I'll mostly focus on the identification of specific LEGO product. You can think of it as a classification problem. I'll go into that in a minute. There's many languages, of course, some of the main ones, but not only ones, is English, French, Spanish, Italian, German and other languages. But these are the big ones. So the easy part where you can get already a good part of that task solved, about 70% as I'll mention, is just simple string manipulation and regular expression matching. So we have our master data and we have these strings and sometimes obviously a particular string of text, a particular part of text will obviously not necessarily contain Lego product. So in some cases, if it's longer texts, we have to do some sort of scanning, scanning of the text to identify which parts of that text contain a mention or of a Lego product. And the next step where, you know, I'm not going to show you a solved problem here, we haven't solved the problem and there is no solved problem really. We're always trying to improve. It's what I call a self learning system. I'll talk a little bit about that. But a couple of examples, some of them quite easy, some not so easy. If we start from the top one here, this is pretty easy because 71712 is actually a product number. I can very easily match to a master data table. But you have then some abbreviations like lo. I actually don't know what LO is. So you have out of vocabulary words, you have abbreviations. Then you have in the second example some spelling. This is not a very severe spelling problem, but there's an apostrophe missing there in meas. Nevertheless, that's also a pretty easy one because Mias Treehouse is actually a name of a product. It's not as easy as Matching a product number. But it's next to easy. It's almost as easy. Then the third one is what I call one to many matches. Because Dots is one of our product series and the Dots refill pack refers to potentially many, many types of refill packs from the Dots product line. In some cases you have what I call meaningful abbreviations like TS4, which stands for Toy Story 4. So that is Toy Story 4 playground. And then the last one is. Well, it's very difficult really to figure out what that is about. We have some Lego Duplo, my first train, for example, or my first animal set, but exactly to which my first this refers to. I'd say no artificial intelligence will help you solve the problem here based only on this. Based only on this text. So moving on to what we are now going forward with is in some cases in particular, when it comes to, as I mentioned, reporting from our retail partners, we have kind of a human in the loop who then tries to match some of these products manually. Whenever some of these match, regular expression string matching issues fail. So then we could use that as training data to build a model, for example, a neural network. Right. So that would be conceptually pretty straightforward. You clean and tokenize the input, which we do anyways. You encode the output, which is how many products we have. Obviously we have a lot of products, but not all of these products are active right now, recent. So if you look at how many products we have that are selling at any point of time, given our product portfolio, turnover is we have roughly a thousand products at any point of time that are actively selling. So that'll be like a thousand classes. That's not a lot as such. And then you train. Right. However, there is of course certain challenges with this approach. As I said, just the simple stuff, string manipulation and regular expression brings you about 70% of the way. So that's not too bad. But I mean that leaves still 30%. And if we want to do this really more at scale, that's not good enough. So we want to. I don't necessarily have a target, but we definitely want to go beyond 70%. Working with dictionaries and sentiment in non English languages is obviously more difficult. There's not so much work being done. There's not so many dictionaries in particular with certain sort of sentiment tags attached to them. The third point is I said we have about a thousand classes, but we don't have that many labeled examples for each of these class. So you're either going to have cases where some classes have no examples at all. Or it could be very, very imbalanced. The types of, you know, the labels can be very unbalanced. And the fourth point is that the classes are not fixed. Like normally with a classification problem you have fixed classes, right? It's a dog, it's a cat, or it's a bird or whatever, and that's it. Or it's a thousand things, but it's fixed number of things. But we obviously always introduce new products and other products get out of the portfolio. So there's new products coming in which initially of course have no labeled examples. Then the last point is, should we be building, let's say one model per language, or should we pre translate into English, for example, and then use an English model? So these are all kinds of things that we are thinking about right now. Now that being said, my field of work at LEGO is not in the area of natural language processing. Just to say that I'm working mainly on recommendation engines. That is kind of where my current work has taken me. But I worked on a number of different projects over time. Also at Siemens Wind Power, I worked a lot with machine data, machine generated data, sensor data. So a modern wind turbine has thousands of sensors that report data at very, very high frequencies, measurements. So I worked a lot with that. I've worked with predictive maintenance, modeling, so trying to foresee when something's gonna break in a machine before it breaks based on various measurements of various sensors and obviously with a lot of time series back in my PhD work and during, during my time at Orcus University. So. But yeah, my main point is that if you kind of have the fundamentals in terms of statistics and in terms of knowing how these models work, obviously there's specific things. Like for example, when we talk about natural language processing, sequence models play a much bigger role than let's say if you want to do image recognition. So there's obviously specialized techniques and specialized models, but if you have the right foundation, I think around the right fundament, then you can really branch into various areas of AI and deep learning. Yes. So I think that's what I wanted to go over with you. And yeah, we're now 15, 16 minutes in, so I'm gonna stop sharing now. Then I'm gonna go over to the Slack channel and have a look at what kind of questions you have and what I can answer. Just give me a moment, I'm gonna try to take it from the top. And also somebody asking if I, if I speak Danish. Yes, I do speak a little bit. I've Been in denmark for about 12 years now. It's not my strongest language, but yeah, I do speak a little bit. So there were a couple of questions around a career in the field, both from Rituja Chowdhari and also I think Sachin was asking around, what would you suggest a student currently in high school or bachelor's to pursue his higher education? Now, since my days in higher education are a few years back and I know that in the meantime the number of programs in data science and machine learning has really exploded. Obviously when I was at university there was no such majors didn't exist. So I did econometrics, which I guess comes to some extent close to data science. But econometrics and statistics of course can be also quite different. What I would recommend, I'd say the quality of course of these data science majors is very difficult for me to judge. But as I mentioned before, I find it at least for myself, having the fundamentals, having a strong background in statistics and in probability theory in particular, and in mathematics in particular, I think linear algebra and calculus, if you can do that, if you can, if you understand that, well then a lot of what is currently at least state of the art in machine learning, we're still doing a lot of calculus, a lot of derivatives, gradients, gradient descents and all of these kinds of things. So I think understanding calculus and linear algebra, matrix operations is quite important, I think to be able to really then broaden your, your skill set to various areas of AI because obviously AI is as I mentioned, you have images, you have text, you have all kinds of data, structured, unstructured, you have like things that I've been doing with the predictive maintenance, survival, survival analysis, which I was doing as well, a lot of different fields. But I guess again the baseline probability theory, statistics, calculus and linear algebra to me is a very strong fundament. And I'm pretty sure that some of these programs of course are teaching these subjects. But I cannot really judge which are good programs or which are not. Plus I'm a little bit, of course, maybe biased. I'm on some advisory boards of some of these programs. Then there are. So Professor X has some questions about topic modeling. Now, topic modeling is not something I've been doing a lot. I've been doing a little bit supporting some colleagues, analyzing some text around employee service. I've been mostly doing lda so latent directly analysis. And I think choosing the number of topics is definitely a tricky thing. I'm doing it iteratively. I'm also looking also whether the Topics and the words that you get into these topics today kind of makes sense to a human. In the end, do they kind of cluster into something meaningful? There are some techniques around how to find the number of topics in some sort of optimal sense. It's a little bit similar like when you do K means clustering, what is K? So you need to choose K. And there's some heuristic methods like the elbow plot where you are looking at the within error, basically squared error within the clusters. And at some point, obviously as you increase the number of clusters, that error will only go down, but there'll be like a neck or like an elbow point on that graph. So you kind of plot that error against K against the number of clusters and then you're kind of looking at okay, where does it really start flattening? Where's the elbow? And I've seen some similar things with around topic modeling where you can look for example at how the perplexity metrics and perplexity is something that is covered in this visualization. So you can look at how perplexity changes when you increase the number of topics. And again you would inspect the graph and see where it kind of elbows out. I think I've seen some other work as well. It's been some time ago now. I'll have to look it up again and maybe I'll try to post there. When it comes to the domain specific texts, I don't know. Even if you, if you have these type of embeddings for these biomedical terms that you mentioned, I would definitely start first. Simple. I think TFTF IDF is definitely something that is easy also to understand and communicate. Because in my experience also and in my job, the communication bit is very, very important. If I have to go and explain to a business stakeholder why these clusters are as they are, I can intuitively explain what is a term frequency, inverse document frequency. Basically I can say, well, if a particular word is appearing very much in this document, but it's not a word that appears a lot, then you know it has a high weight. With embeddings it gets obviously more difficult. So my approach is always start as simple as possible. Yeah, And then there's a. So Shubham Junger is asking around abstract summarization of a text and I think that's something. It's not something I've done in practice. I think it's a, it's a difficult topic as far as I understand. It's one of the frontiers, I think when it comes to natural language processing. I Think what apparently seems to be successful is sequenced. So it's a bit like translation. So it's like translating, let's say, English to French. Like we have examples also in the specialization, but you're translating practically a longer text to a shorter text. So you have a sequence to sequence problem and that you could try to solve with LSTMs with some of the new approaches like Transformers. So I think that's probably the cutting edge, I guess, in that field. But I don't know much more really. I cannot say that I've been working and applying these types of things, these types of models. Somebody suggested maybe LO stands for Ninjago, maybe. I don't know why LO would be Ninjago, but it could be. Yeah, That was Jason, Jason T. Brown suggesting that. Thanks, I'll look into that. And I mean, again, in that particular case, it was very easy for me to identify the product. But I'm still curious what this LO stands for. Then there was a question around. So, Elkin, David Diaz Plata. Is it possible to do a recommendation system to recommend an employee to a specific task? That's a very interesting one. Yes. I think that you would need to think about what is your optimization metric here. So normally when you do a recommendation engine which is customer facing, you optimize on some sort of metric. It could be click through, it could be how many comments or likes somebody leaves to think how well they rate the products. It's typically products, right? You recommend products, you recommend songs, recommend movies, and then you check afterwards whether people actually click on them, like them, watch them, listen to them, and things like that. So when it comes to employee to a specific task, one has to be a little bit careful. Do you recommend something matching their skills or do you recommend something that they would like to do? Ideally those two should match, but. But yeah, you have to be a little bit careful what kind of metric you're optimizing on and then of course, what kind of data you have. If you can collect some data where you have a lot of employees rating types of tasks, similar to the Netflix data set, that's one of the most famous, I guess, data sets in this field. So in the Netflix data set you have movies and you have people ranking these movies. So you could have people ranking tasks and then you could run something like ALS model, alternating least squares, which is very common for these types of sparse matrices because you would not have everybody ranking every task. So you would run like a ALS model on that rating matrix and you would be able to come up with some sort of, you know, recommend their system. Yeah, I think it's, it's possible. Moving on to a question by sriq. What is the role of platform language like Python? R TensorFlow. Well, I mean I'm a heavy R user, I have to say I love R. I also love Python. I'm definitely not on any side of that, of that as a statistician. I, I really, you know, I did a lot of matlab back in the days, both in my academic work but also at Siemens Wind Power. My main tool was MATLAB. Honestly, I don't care. I think for me TensorFlow feels more at home kind of in Python, but I can do the Keras has more or less full functionality of Keras in R as well. So when I was doing some recommender models, for example, there's very powerful parallelized libraries in Scala and Spark. So then I did the same like you could use an interface from into Spark from R, an R API. You can also use a Python API and the syntax is almost the same and I was in fact comparing speed and performance and it was also the same. So I spend a lot of time comparing Python's kind of Spark implementation versus Sparkly R package in R and I ended up saying, well it's the same, I just pick one or the other. So I very much like R because of shiny as well. I'm a big fan of shiny and I like to do dashboards. I'm not much of a front end person web design and this and that, but just shiny makes it so easy. So I'd say, you know, probably given the disposalization obviously is in Python it's probably. Many people say that's easier to do. For me R is easier because that's where I started. But I think just pick one and go ahead, you'll probably find what you need in any of them. And of course then there's TensorFlow, there's PyTorch. Yeah, I haven't been using Pytorch much. I think for me TensorFlow and Keras offers pretty much what I need within deep learning. Okay, so I think we have a couple of minutes more so I can keep going. Lda, sorry, I'll go for Cyan is next one. Do you use time series analysis in the Recommender system or in nlp? Not classical time series. So what I mean, classical time series is things like autoregressive, integrated, moving, average, so Arma, Arima, ReMax and variations. You can also do time series with random forests by the way, which is a little bit weird but you could do it. So for the recommender systems, not currently I'm not using any sort of time series analysis. In nlp, you obviously use sequence models, which is a kind of time series. Because when I speak now and when you see a written text, it comes in order. It's not time series, maybe in the classical sense that you have some sort of timestamp, but it's a sequence of events or a sequence of words. So then yes, you kind of use time series, but again, not the classical time series analysis. Even though in theory, now that I'm thinking about it, you can obviously, once you tokenize your words, you could try to run like a classic ARMA model on them. I don't know. I'm gonna learn much though, because those tokens, it's not like a stock price, right? A stock price value of 20 has a meaning. The stock is $20 and then it's, it's 15 or it's 25. So these, these changes in stock price have, have a meaning in, in language modeling. I guess it's a bit more tricky like what happens when you move from 3 to 17. I don't know. Haven't thought about this really. Analysa Alda is not used for topic modeling. I think it is. I've used it. In any case, it's one of the tidy text. For example, package in R has LDA for topic modeling. If I understand you correctly, You can pick the topic with high coherence. Yes, I find understand correct in terms of number of topics. Yeah, possible. But you still have to decide, I guess at what point you want to stop when it comes to number of topics. And I think to some extent is also again talking about the number of topics. First of all, there's some heuristic methods, but again, to me mostly they kind of have to make sense, right? If they don't make sense, I'll try to change some parameters and maybe change the number of topics, reduce, increase, try to find something that is a little bit more meaningful. Otherwise, I think it's again, it's very difficult to present or to explain to somebody why these topics occur if they don't kind of make sense to a human. Because in particular with topic modeling we're very much interested in finding meaningful topics. It's not so much like an image classification where I don't care if you understand the deep neural network. If it classifies well, then it works well, that's great. But here it's really important that people understand what these topics are about. Sven Chilton is Asking if I know any good resources for multilingual speech or text recognition models. And the answer is no. So I think what I tried to do is try to find some dictionaries. You can find dictionaries at least in some of the bigger languages like German, Spanish, English obviously, but haven't been looking much more into that really. There's a question about GPT3 its capacity to understand and solve arithmetic in some basic programming tasks. I don't even know what GPT is to be honest, so I guess I cannot answer that question. Then there's a question around string manipulation. So that's Sujay asking about string manipulation regex methods working about 70% of the time. Yes, I mentioned that, at least for the data set that I was working with. So how would you go about building a system that will recognize that it made a mistake and redirects the examples? Yes, so I think some of that. So when I say that it solves 70% what I mean by that is that I'm pretty sure that they're correct like 99.99%. In particular the cases where I match a product number, these are hundred percent correct because there's no ambiguity really. Some of the text based. Matches obviously can be wrong and I think again there without the post analysis of kind of where things go wrong. So you would have to look at the examples that go wrong and potentially somebody I think also might need to label those examples as being wrong and then giving them the correct label and then over time that's what I mean. It's a self learning system that over time it will learn to correct itself from these examples. But I think it's difficult to go about it without really spending some extra time on these cases where like you would do. I think with any model which makes mistakes you would spend more time on trying to figure out why these particular mistakes are happening. Oh, some good papers. Surab Tamle is asking about some good papers for time series analysis. Oh it's a huge field. I mean not papers I can tell. I can suggest textbooks in particular when it comes to the more classical time series analysis which is what I've been trained with. There is some pretty good textbooks from, from the econometric perspective, you know, talking about things like stationarity, unit routes and all of these, all of these concepts which are quite important in time series analysis. I would start from. To me in general, the bible of econometrics is a book called Econometric Analysis by a guy called Green. His family name is Green. I don't remember his first name. And there is a chapter on time series analysis as well as things like of course there's linear regressions, logistic regressions and it's an econometric book which is why I think it's very different from most of the books in machine learning. So you will learn about what is maximum likelihood, how to do Lagrange multiplier tests and likelihood ratio tests and try to understand what are standard errors of coefficients in the linear regression, why they are like what they are Just talking about this now. When I first started doing exam machine learning or deep learning I was very surprised if not shocked to see that linear regression is being done using gradient descent because the solution to linear regression is analytically known. But of course you need to with large data set it involves inverting a very large matrix which of course is a very expensive operation if you want to do it in practice. But I'd start with econometric analysis. Look at the time series chapter depending on which level you are really that is where if you really want to get into the field of what is a time series, what is stationarity, what is unit root, what is an integrated process, what is an autocorrelogram and these kinds of things. Yes. Next question by Bo Wenchen I guess Shapley values. I guess you mean I haven't done it. I know some of my colleagues are looking into that but apart from being in a presentation when they were showing us what, what those actually mean, I cannot tell you much more on that. I think it's something around explanation of a kind of factor attribution. When you have things like random forests and models based on trees practically, then you can use these Shapley values to find importance of your features. So which features are important? Whether it's a regression or classification, I think these methods are more suitable for classification but they can also be used for regression as well. Yes, Nancy, that Scott has given a link here to the GPT paper. I'll have to have a look at it. Yes, because that is. I'm not, I'm not familiar with that one. Thank you very much, Scott. So I think yeah, currently there's not too many other questions. I hope that some of the answers kind of made sense to you all. And thank you very much again for joining today's event. I think the channel will be up at least for some time. So if there's anything further again in particular things around time series analysis recommendation engines is where I think I have maybe relative to other people here in this specialization. Other mentors might have more knowledge then feel free to ask. Maybe just one last one. I'll take. And I think I'll take the rest offline. Yeah. So some. If you're new in the field of nlp. So that's Pau. Ravi. I'd say. Yeah. I think you're in the right place in this specialization. I think you'll get a lot of different ideas for different projects. You can do a lot of things right. You can do translation, you can do sentiment. I think sentiment is an interesting one. Is what a lot of people start with. I guess the classical example is sentiment analysis of tweets. That's like what everybody. It's like the hello world. Almost like a fennel piece. So that's probably what I would do. And I'll try to give you some suggestions where you can start there. Thank you very much, everybody, and enjoy the rest of your day. For me, it's almost 8pm now, so I'm gonna go and rest now. Put my kids to sleep. Yeah. Thank you very. Much.
