Ask Me Anything about Natural Language Processing (NLP)

Sam. Foreign.

Hi everyone. My name is Ishita Chaudhary and I'm a Product marketing manager at DeepLearning AI. Welcome to Ask Me Anything About NLP. A special thanks to Hugging Face for co hosting the event with us. I'm thrilled to announce, for those of you that don't know already, that last week our Natural Language Processing specialization was updated with brand new lectures, labs and assignments in course four. We also partnered with Hugging Face to provide even more enhanced experience with Transformer model applications. So if you haven't yet, check out the updated specialization. There's a link in the description below. It's great to see so many viewers from all over the world join us today. Why don't we start by sharing a little bit about yourselves, where you're from, what you do, and if you've already taken or plan to enroll in NLP specialization. Hi guys. Hi Victor, Michael Shakeel. Nice to see everyone here. Okay, please keep sending these messages. We love to know where our viewers are from and what they're doing. And if you have any questions, just write those in the YouTube live chat. While you do that, I'm going to take this opportunity to introduce our guest speakers for this event. Eunice Bensuda Muri, the instructor for our NLP specialization, and Lewis Tunstall, a machine learning engineer at Hugging Face. They are here today to answer any questions you might have about nlp. To post questions for our speakers and to discuss about the NLP specialization with your peers, simply go to the Deep Learning Online community on Discourse. For those of you who are new to our community, let me show you how to participate in today's event. Great. If you're on our YouTube page, just scroll down to the description below. If you haven't yet joined the Discourse community, please click on this link and create your account. If you've already created your account or like you're already part of our community, simply click on this link and you will be directed to the Careers page on our Discourse community. Here you can ask a question by clicking on new topic and just typing what you want to talk about. As you start typing, anything similar questions will pop up on the right hand side. So if you know if like someone is talking about a similar topic and you can see if their question kind of covers your question or if your question is unique and you want to go ahead and post it. So this is how you create a new question. You can also go and vote for other questions. To vote for other questions, simply click on a question and click on Vote. If you want to see the most upvoted questions you can click. You can sort this entire list by votes and see how many votes each question that other viewers have asked have received. Yeah, I think I can stop sharing my screen. So guys, today's event is about you. It's about you getting help for all your NLP related questions. We encourage you to go to Discourse to join the community and and to ask any questions you might have and also what other questions are interesting for you. I hope that was helpful. If you have any more queries, please ask in the YouTube live chat and our moderator will help you out. Okay, without further ado, let's welcome Eunice and Lewis to kick off our event today with the introductions and a little bit about the NLP specialization. Welcome folks.

Hello.

Hello everyone.

Thanks. Yeah, thank you Sita for organizing and Alice. So I'd like to start by sharing my screen one second. Perfect. All right, perfect. So yeah, thanks for organizing. Thank you Louis for joining us and for a hugging face to for having partnered with us. So I'd like to start today by talking a little bit about natural language processing. Before I get started, I'll just give you a very quick background about what I usually do. So I did my math, my my bachelor's in math and computer science and master's in statistics from Stanford. I'm currently in adjunct lecture of computer science over there and I teach the applied machine learning class and help build the deep learning class over there and taught another class called Teaching AI. Previously I worked on the deep learning specialization. Maybe you're familiar with it if you already took it, but basically it's a suit of five courses on Coursera about deep learning. I also helped work on the natural language processing specialization and previously I worked as a software engineering as a software engineer in a variety of roles in the past. So why natural language processing? First of all, natural language processing is one of the most sought after skills in the job market today and it deals with handling a lot of the text data. So anything that you see in text, from, I don't know, from newspapers to articles to novel, this text, it can also combine some speech and you combine these two to try to understand what is going on and to create sense out of these two inputs and then you could respond. So for example, in the case of a chatbot, you might respond to a certain question or in a question answering context, given a question and some text, you might be able to identify what the answer to that question is. So basically natural language processing is handling all of the information and data around you and trying to make sense out of it. So that is why we went and built the natural Language processing specialization. So it is a specialization which consists of four courses. And the first course is called Natural Language Processing with classification and vector spaces. And I'm going to talk a little bit more about each course and then I'll also talk about what we changed in the refresh. Course two is about probabilistic models and natural language processing. So like how you can use the statistical models to come up with cool algorithms you can use. Course three is about sequence models and it handles everything from RNNs to LSTMs to GRUs to Siamese networks and so forth. And then course four is about attention models and this is where we talk a lot about the transformer. We go over machine translation, summarization, question answering and chatbots. So to start with, in course one, you start by working on classification vector spaces. So you start by building a sentiment analysis model in logistic regression and naive Bayes. And then you proceed to learning about vector spaces. So how do you represent a text? How do you represent a word? How can you represent a tweet? You also actually manipulate word vectors so you can predict countries of certain capitals, which is really cool just by manipulating these words vectors. You also learn about machine translation. So for example, how you can translate one word to another word by learning the simple mapping. And you also learn about k nearest neighbors and locality sensitive hashing to speed up the search for the nearest neighbors, which is something you will see later. In course four, it helps you with the attention in the transformer network in a different model called the Reformer. So course two is about probabilistic models. And in probabilistic models, the first thing we go over is autocorrect. Autocorrect of course is a very useful tool that you use every day. For example, in your phone, when you send a text message, you use autocorrect, and that's really useful. Other thing you learn about is called parts of speech tagging. So this is important also in speech recognition. And we use the Vitrebe algorithm and dynamic programming to teach parts of speech tagging. For example, reason why it could be important is for example, if someone inputs a query like book of flight, book is a verb, but if someone tells you give me the book, then book is a noun. So depending on the parts of speech tag, book means something different and it could help you to answer whatever that query is. We also go over autocomplete. So you learn how to build an autocomplete system with an N gram language model. And you use autocomplete in your emails every day in your text messages. And it's an important tool that you can use in your own applications. The last thing we cover is word vectors. And although many people usually just download word vectors online and they use them, we thought it would be nice to show you how these word vectors are actually trained. Just to give you an intuition of how things are working. We, we use continuous bag of words model here. But there are other techniques for word vectors that are more sophisticated, but it just helps build the foundation for word vectors. The other course is the third course, sequence models. So in sequence models we do go over sentiment analysis in more detail. And it's. You use the sequence models to capture, you know, for example, if I told you if only it was a great day, we would have had so much fun. With the initial models, you might not be able to capture that it's a bad sentiment because it has, you know, a great day and it has so much fun. But with these more sophisticated models, you can actually capture these sentiments. We also go over text generation to help set up for question answering and chat bots and machine translation in the next course. So we show you how text generation is done. We go over named entity recognition. So for example, here Sharon flew to Miami last Friday. You can see that Sharon is a person. Miami is a geographic indicator and Friday is a time indicator. And then we go over Siamese networks. And this is one of my favorite topics in the specialization. So given two questions, instead of training a classifier to identify like K plus 1 classes, given a new input, you basically just compare whether two inputs are similar or not. So given a question and another question, you can identify whether they're similar or not. The last course is attention models. And over here we start exploring the transformer network. So you start by building a machine translation system from scratch. And then we show you how to build a summarizer with these transformer networks. And then we go over question answering. So given a question and some text, perhaps find the answer given the question, what is the answer to that question? And then we conclude by showing you how you can build a chatbot. So each of these programming assignments is a little project that you can see how to implement such a product and the refresh of the focus. So a lot of the work was actually done in the transformer network. We also added several labs, we added quizzes at each week. At the end of each week we added lecture notes. And also since we launched towards the end of last year. We got a lot of feedback, so we incorporated a lot of feedback in this refresh and we also partnered with Hugging Face and I'll let Louis talk a little bit more about Hugging Face. So with that said, thank you and I'll let Louis take over.

Awesome. Thanks, Eunice. And thanks to DeepLearning AI for this invitation. I'll share my screen now. So let's see. You should be able to see this, but if not, please interrupt. Yeah. So what I'd like to talk to you about today is a sort of brief tour of the Hugging Face ecosystem, because in the deeplearning AI specialization, you're going to be training Transformers and potentially even sharing them with the community. So I thought I'd give you a kind of high level picture of what Hugging Face does and some of the kind of exciting developments that are kind of on the horizon. So to get started, I'd like to talk to you a little bit about myself. So at Hugging Face, I'm a machine learning engineer in the open source team. That means a couple of different things. On the one hand, I help contribute to the open source libraries that Hugging Face produces. You're probably very familiar with the Transformers library, and you'll be learning about this in the deeplearning AI course a lot. But Hugging Face also has other libraries like the Datasets Library, which is high performance Processing library for simplifying the analysis of data sets in NLP and in computer vision and in audio. So that's the one side of the sort of, say, software engineering side of things. But then there's another side which I'm very passionate about, which is sort of the educational component of what Hugging Face does. And you may also know that Hugging Pace has its own kind of online course you can take, and this is exclusively focused on Transformers. So when you've done the deeplearning AI specialization, and if you want to really go deeper and deeper into this topic, I recommend you check that out. And I've also been working on a book with Tom Wolf and Leandro von Wera, which is basically again, about NLP with Transformers. And this will be released sometime early next year. So if you don't get enough of Transformers today, you might be interested in checking out both of those resources. And just to mention that we have a sort of live event of our own coming up in November 15th. And this is where we're going to also launch the second part of our course, which kind of goes deep into the sort of NITTY gritty of the datasets library, the tokenizers library, and even how to train a transformer from scratch. So if you're interested in taking part in this event, I recommend you go to this link and then you can find all the details on how to register. Okay, so I'd like to tell you a little bit about kind of the motivation behind what we do at Hugging Face. So I myself was originally working as a data scientist for a Swiss telecommunications company. And one of the kind of common problems that you face when you're working in industry is that sort of every day there's like some new state of the art model on the archive or on Twitter and you say, oh, it'd really be great if I could get my hands on this model. And for a long time it was very hard to sort of take these kind of research papers or these research repositories and kind of implement them yourself in your own projects. And there's been some kind of positive changes in the last few years. So one thing that the archive has started doing is providing kind of links to code and the model weights. So, so in this case you can see the attention is all you need paper, which is the original transformer paper. And here there's a link that tells you go here if you want to get access to the source code. And this is a great step in the right direction. But it has a kind of problem which is that you may be aware that there's currently thousands and thousands of pre trained models available. And kind of figuring out where do you start? Repository should I start from? Is not so easy. And the other aspect is that every kind of research lab or even independent researchers, they all have their own sort of idiosyncratic way of writing code. They might have their own APIs, maybe there's no documentation. And a lot of the time in machine learning and in deep learning in particular, there's kind of like a reproducibility issue, which is that even if you're given the exact same source code, it may not be obvious how you reproduce the same results in a paper. So the kind of question at Hugging Face that we sort of ask ourselves is can we do better than this kind of very disparate and kind of distributed way of having code all over the Internet. One thing you may know of is the Hugging Face hub. This is the sort of central website for Hugging Face. And on the hub you can find pre trained models for transformers, but you can also find models for other domains. So we now have models that support things Like Audio, we have models from Spacy and we're slowly growing more integrations into kind of like the open source domain. So you're probably going to see, for example, scikit Learn in the future and things like this. And the thing I want to sort of emphasize is that hugging face has always traditionally been associated with nlp, primarily because of the Transformers library. But what we've seen is that the Hub is a great place where people can collaborate and share their models. Not just for nlp, but for machine learning in general. If you visit the Hub, you'll find there are now computer vision models and things like this. This is where we're going as a company. I just like to end with a kind of quick overview of the ecosystem from the open source side. So as I mentioned, we have the Hub, which is like the central place where we store models, data sets, metrics and so on. And on the open source side we have a couple of different libraries, some of which you'll be playing with in this course. So there's of course the Tokenizers and Transformers library, and there's also the Datasets library, which I mentioned before, which is a really easy way to kind of load data sets of very large size and process them efficiently. But we've also got libraries like Accelerate, which is designed to do your own custom training on distributed infrastructure. If you don't want to have to change your code every time you switch from a GPU to a tpu, then this library is a great place to start. There's also Hugging Face Optimum, which was just released a couple of weeks ago, which is a set of tools to make your Transformers efficient in production. Here we're talking about things like quantization, pruning and so on. The last thing I'd like to mention on the open source Hub side is something called Spaces, which we're going to launch publicly quite soon. This is basically the infrastructure that allows you to deploy your models and build interfaces to them using Streamlit or Gradio, which is a very simple way of sharing your models with your stakeholders or with your friends or even your parents. Then on the commercial side we have several products which, which we provide to customers. So there's AutoNLP, which basically provides an interface that lets you train and deploy transform models with no code. You just have to upload your data, click some buttons, and then it will train the models for you. And this is great if you just want to build a set of baseline models, for example, for your project, and you don't want to Spend tons of time checking which different hyper parameters you should use. Then we have the Amazon SageMaker partnership. And so this means that on AWS you can find some dedicated deep learning containers where you can run SageMaker with hugging face integrations. And this again lets you train models and deploy them with very little boilerplate. The last two things to mention are the Inference API. This is basically the back end behind the hub, which allows you to generate predictions from your models and feed them into your spaces or even into your own applications. And then there's Infinity, which was the most recent product that we've released, which is an optimized container solution for deploying transformers and getting very, very fast latencies. So we're talking about things in the realm of like 1 millisecond latencies for your transform models. And so this is a great product for people who are working in companies where they have very large loads or they have real time needs. And this, this product will optimize it for you. So with that, I'd like to sort of end this very brief chat and maybe we can dive into some questions.

Yes, absolutely. Thank you for your present for these amazing presentations. Lewis and Lunas. Eunice. Lewis, it was super helpful for me when you talked about the ecosystem because I was just like, oh, I don't know so many things. And this really helped kind of clarify that. And you're right. I think it's time to look at and answer some questions that our attendees have for us. So I'm going to share my screen so we can look at this course. Let's go. Okay. Can you all see my screen?

Yep.

Okay, great. So let me sort by votes.

Okay.

So this is a great question by Jean. What would be the roadmap for an engineer who is starting the career in ML and wants to specialize in nlp?

Do you want to take that? Eunice and I join after.

Oh yeah, that sounds good. Yeah, I mean, definitely a good question and very relative. So I would say first, start by learning some basic coding skills, general for loops and basic coding. And then there are some good resources out there. So there is a course actually on Coursera, a machine learning course by, by Andrew Ng. Then there is also a deep learning specialization by deeplearning AI, which actually goes from like building blocks all the way to like showing you at least like General, you know, CNNs and sequence models, and concludes in Transformers. Then there's also the natural language processing specialization that I just recently spoke about. And I would say also the hugging phase course, if you want to Go more into Transformer networks. So this is like basically in a nutshell. Louis, anything else?

Yeah, I would just like to maybe add that I actually learned deep learning from DeepLearning AI. Seriously. The very first deep learning course was, I think when DeepLearning AI went live the first time. And for me personally this was an extremely exciting course because at that point I was working as a, as a data scientist on time series and this is quite a different domain where deep learning hasn't yet kind of broken into so much. And yeah, seeing that, you know, Andrew was basically teaching us how to train, I think it was like you had to build a happy detector, like to open your, your front door, you had to smile and do things like this. This is really cool. But the other thing I was sort of added, and this is always a tricky question because everyone's background is very different, right? And their circumstances can vary a lot. But one of the things that I've seen both being applying for jobs and also when I've been hiring for people is that you kind of have to convince someone who doesn't know you that you can actually do the job that they're advertising for. And in ML, a great way to do this is to sort of show like with something real that people can interact with that you've built something kind of interesting and sort of novel. That's like the key thing no one wants to see yet another LSTM network on Bitcoin. We want to see ideally using machine learning to solve problems that you care about. And this could be things in your personal life, but it could also be things that may be in your job. You see an opportunity to apply these techniques and I think this way you can show people that you can do the task. And it's also a great way to get that really hands on experience. Because so much of the real world of machine learning and data science is messy. It's not like a carefully prepared data set. And learning those skills are valuable.

No, that's, that's super relevant, super cool. That's, that's, that's really interesting. Lewis. So you're One of the OGs, you know, very cool. Okay, we do have a lot of questions, so moving on, our next question is what is the significance of RNN, LSTMs, GRUs after transformers? Do you still think there's a future given that Transformers have outperformed them? And if yes, could you suggest where, like what possible areas you could use them for?

Do you want me to start, Eunice?

Yeah, go for it. Start So I can go after.

Sure. So I think one thing that you often experience in real world NLP projects is you don't just care about like performance on a benchmark, like on some academic benchmark. You also have constraints about, for example, the size of the model and even maybe the latency. So sort of how fast can the model generate predictions and how fast can you train it? And one thing that I've personally experienced is that you can train some LSTMs much faster than you can train some transformers. And so in those cases, if you're just trying to get a very strong language model, for example, using something like ULM Fit from fastai would get you results in the space of an hour or so and maybe the equivalent transformer would take much longer. So I see that's the kind of personal place where I see LSTMS being very relevant.

Great. Eunice, do you want to add something?

I think Luis nailed it. It's basically just as he said. There are other factors that you have to keep in mind like size of the model, how fast it performs, and also LSTMS can also still be useful in dru, so it's good to have both in your toolbox.

I actually just remembered one thing. Today or recently Google published a post talking about how they implemented like I think kind of grammar correct on device and there they have a transformer as the kind of encoder, but the decoder is an lstm. And they talk about in this blog post that they had to do this because of those exact reasons. So you know, if LSTMS are good enough for Google, I think they're good enough.

Yes.

Very cool. Okay.

Right, so our next question is for someone who's already done the NLP specialization, which course in this updated specialization be more relevant? Would it be course for Eunice? Do you want to take that one?

Yeah, definitely. If you already, if you've already done the NLP specialization course four, we changed all the lectures in weeks one and week two two, and then in weeks three we added hugging face content and more about the transformer. So a lot of the content was changed in course four, but we also added in courses one, two and three, basically almost after every video now you have these lecture notes that go a little bit in more depth. We added also quizzes that you can use and a few labs here and there. We also basically took all of the feedback we got and tried to incorporate as much of it as possible. But I would say if you really want to, if you only have time to take just One course, then go for course four.

Yeah, super great. There's been updates across the board, like across four courses, but there's a bunch of new content in course four and also in course three. Correct me if I'm wrong. So yeah, if you, if you've already taken it and you just check what we would recommend, check out those courses before you go for like course one and two, which also, as Eunice said, has lots of new lecture notes and content and just labs. All right.

Sorry.

Let me just check if there's like new upvotes.

Okay.

When should I create or train a custom tokenizer? I think there's a little more. This is a long paras if you guys want to just quickly read this.

Yeah, I could also just maybe start by talking about some general things here. So the, there's like, if we take a step back, if we're talking about say transform models, the sort of modern paradigm is transfer learning. So you take a pre trained model, so something that has been kind of trained to predict for example the next word in text for say Wikipedia. And then you've got something that is really good, like an autocomplete. And then you take this model and then you typically apply it to some downstream task that is quite different. For example, text classification and the way that transformers and in fact almost all NLP systems process raw text is not actually as strings but as kind of vectors or numerical representations. And the process of going from text to these vectors is through this thing called a tokenization or a tokenizer. And that process of converting text into tokens into vectors is kind of learned as well from the data. So it's not learned in the kind of conventional gradient descent way, but it's learned through like statistical properties of the corpus. So the question is kind of asking, you know, when should I worry about having to create my very own tokenizer? And the general answer to that is if your domain is very different from the domain that the model was pre trained on. So if I have bert, which was pre trained on Wikipedia and some books, but my domain is like maybe, you know, protein sequences or something really different, then the tokenizer of BERT will actually do a pretty bad job at tokenizing these protein sequences. And so then you should really create your own tokenizer on your own corpus. But the downside or the catch is that then you will need to train your own model. In general, I think that you can do some hacks to stitch together different tokenizers with different weights. But generally speaking, as far as I know, maybe Eunice can correct me, you'll have to then do the actual pre training yourself.

Yeah, exactly. Yeah, basically. Exactly what Lewis said. If you are just basically retraining on a completely different domain, then you have to retrain from scratch.

Just one caveat that occurred to me is that if you for example, have emojis in your corpus and maybe Bert didn't see so many emojis because there's not maybe so many in Wikipedia, then I think what you can do is you can add them as special tokens. So you can kind of say my tokenizer has some vocabulary and then I'm going to add some new tokens to the vocabulary and then I can just kind of resize the embeddings that I feed into my model with these tokens, but they're going to be kind of randomly initialized and so then I'm going to still have to kind of learn those representations of those emojis somehow.

Sounds good. Sorry, do you want to take this question which is are there some issues that might appear from using pre trained tokenizers?

Yeah, so one thing that will happen is like if the tokenizer is not adapted to the to the domain, then it will actually chunk the sequence into lots of different tokens. For example, if I have again the protein sequence example which is lots of letters like a, G, T, A and stuff, then something like the Bert tokenizer will say, oh, I don't really know what these words are, so I'm just going to chunk them into the smallest units that I have, these subwords. Then you'll get a very long sequence as your input to the model. And the problem with that is that most transform models have a limit on the input sequence length. And on top of that, most of the attention mechanisms are order N squared in complexity, which means that you're going to pay a price both in time and potentially you won't even be able to fit it into the transformer.

Got it. Okay, cool. Let's move on.

Sorry. Okay.

Oh, I think we have another top question. How far away are we from using NLP for fact checking and for logical fallacy recognition? And I think this person Lewis is interested in helping recognize fake news and understand bias and logical fallacy. And how can we create something with like Grammarly add ons and do we. Do you know if something like that already exists?

That's a great question. I think Eunice can go first.

Yeah, I mean there is definitely lots of bias that's going on right now in nlp. So I don't know, maybe I Think we have it, it's somewhere in the NLP specialization where you go, you know, doctor is two men and then, you know, nurse is two women. So you have all of these like a lot of bias going on. And I think if we want to fight this type of bias, ideally we want to like, make sure that the bias is not as in our, is not in our data sets. And that's going to be some good amounts of work or at least we should make sure that the, the engineers working on these types of models are actually fact checking these, these data sets. So that's in terms of like, you know, bias over there. As a Grammarly add on. I know there are like a few amounts of companies, I cannot remember their name right now, but who are working on like Grammarly plugins to recognize fake news. It's, it's a little bit hard, but who knows, maybe it will come at some point. I think there's also, there are a few nonprofits who are trying to tackle exactly this type of topic. I'm not sure if that's answered the question, but maybe Lewis can add a little bit.

Yeah, I mean, question.

But maybe Louis can add a little bit.

Yeah, I mean, I totally agree with you. I would sort of add one sort of small caveat on the bias question. So one thing that has, I don't know if people know. There's something called the Big Science Workshop, which is this very large community event to train a GPT3 scale model from scratch. And one of the discussions that I've seen in this collaboration has been around whether you can essentially retain bias in your data, but then have a way of controlling the outputs of the model with some, for example, prefixes. So there are some transformer models where you can kind of provide them with a prompt or an input which kind of is a prefix that says like an instruction. Don't output these kind of things. And then as long as the model's been trained in an appropriate way, even though there's bias in the data, you can kind of tune those outputs a little bit. But this is definitely a sort of very big open question in the community. And I'm not sure that we can really say we have solved this in any, in any sense.

Okay. We're getting such good questions and we have such amazing answers from you, but I have to like, I think we have a lot of questions, so I'm trying to make sure we answer as many learners as possible. Okay. I think we've answered all of these key challenges in NLP that seems Like a pretty broad question, just like quick, quick thoughts on this. Besides. So it says, besides explainability, what are the challenges are significant in nlp? Any like quick thoughts?

I think other than explainability? I think the ethics behind the NLP models is a big one. Just making sure that we're deploying fair models, that's one big challenge. Another big challenge I see is that a lot of the current models make use of transfer learning and they require huge amounts of data to train and as a result it requires massive amounts of resources. So maybe individuals would not be able to train, I don't know, DPT5 or GPT3. So that's another big challenge. So maybe, you know, big companies will probably have access to the better models because they have more resources. That's another challenge. So I would say just at least where I see the field going is there's a lot of transfer learning going on and a lot of training of these massive data sets. But thankfully, you know, hugging faces is hopefully opening up that space and allowing individuals to use these big models.

Yeah, the other thing I'll just add is, is actually like the language barrier. So unfortunately most NLP research is in English and most of the focus from researchers are on English data sets. And so if you for example, want to, you know, apply NLP in, in a non English language, there's a pretty good chance that some such a pre trained model like BERT doesn't exist. And so then you're kind of in, you don't get to benefit from all of, all of the work that has already been done. And I think figuring out how to bridge the gap between, you know, very common languages on the Internet to others is an open challenge.

That's honestly like a really good point. We recently had a learner reach out to us and they were doing some NLP work in Indonesian because they couldn't find those libraries anywhere. So I was, I was quite excited to hear about their projects. Okay, moving on. Which companies need NLP developers and what skills should we have in advance as NLP areas are growing each day? Any thoughts on this?

I think the simple answer is all companies maybe. I mean, I know, I'm joking but like at some other level, right? We use text in business like for everything, right? It's emails or internal documents, we have our own wikis, all these kind of things that NLP can really help with. And I would say from the skill side, it's very similar to what Eunice told us at the start about, about you have to have some kind of basic coding Skills and also having a kind of good understanding of the fundamentals of nlp. And the best way to do that is to, you know, do these courses, but then also build your own projects. I think those. That that combination is a great way to become proficient.

Solid advice. Let's go ahead. Oh, that is a long context for the question. Predicting using train transformers, sequential slow process. Sorry, I'm just trying to. I'm just gonna. Okay, this is a very technical question. Is this something that we can. Do you guys want to take something like this now or maybe we can reach out and like answer this question later?

Yeah, I think so. It looks like a pretty hard one to explain in just speech.

Yeah.

Okay, we will come back to you and we will get back with some answers, but thanks, sir, thanks for sending in your questions. Okay, how do you implement graph based neural network on NLP entity extraction models? Again, super technical or something you think we can just do an overview of? It's fine if it's super technical. We can let our learners know what the right answer is after this event.

I actually don't know the answer to the question, so maybe Eunice knows.

I never implemented graph based neural network on NLP entity extraction. I just know that I once implemented name density extraction with an lstm. But yeah, I mean, it's just a question of figuring out the inputs and the outputs for the lstm, at least just making sure you have the right tags and the right output. But for graph based neural network, I never did it before, but I guess it's similar.

Okay, Abed, maybe you can tell us or. No, just kidding. I hope that that helped at least close to the solution. But yeah, if you have any follow up questions, please feel free to add it to this topic. Transformers for similarity tasks. I think again, this is a very technical question, so. So maybe we come back to it and if that's okay with you guys.

Yeah, I mean, I guess just having a quick look. They're talking about sentence Bert or sentence transformers more generally. And I can just say a few words about that. So one of the things that's like quite interesting is that when you have a representation of a document, a numerical representation of a document, you can then ask questions like how similar is this document to another document? And you can check that mathematically because these are just vectors in some space and sentence transformers or the sentence BERT paper that they're mentioning here is kind of like a twist on the original transformer that's kind of designed for converting all of the vectors that come out of a single transformer into like a single representation. And this gives you very high quality vectors that you can use to do things like search. So you can take a query from someone, you can represent it as a vector, and then you can check how close that vector is to all the documents that you might have in your database. This thing is called neural search. It's a kind of hot topic right now. And you can also use these sentence transformers for just using them as embeddings that you can then cluster. You can do all these kind of aspects with it. So check out the sentence transformers library if you're interested in diving into that.

Wow, thank you.

That's.

That's super helpful. Sorry, let me just refresh this once. I think we've answered all of these. Okay, Eunice, you wanna, you wanna take this one?

So we actually still use tracks for our course for. We also use hugging face libraries. So I think at least what I felt like when looking at those assignments that we built is the intuition is kind of similar on the frameworks tend to be pretty similar. It's built on JAX and it's like plus like a lot of the times you'll just be like downloading a transformer model, say from hugging face or from some other framework. So like what we try to do in core sports to just basically show you how the transformer network is actually being built from scratch and then you can go and download other libraries. So.

Yeah, sounds good. What do you think about this? Discuss debate on how to create word embeddings for different contexts. I think like Twitter, like posts, small size comments, product description, social media posts. Is it better to use pre trained word embedding trained from scratch to capture the semantic of the particular application or fine tune one? Does anyone.

That's a great question. That actually is pretty related to the thing I was mentioning about with sentence transformers where you're trying to create these embeddings from your text. And I think the general experience I've had is that it's almost always better to train your own embeddings because most of the time your domain will be quite specific. It will be something like, I don't know, like internal customer documents or something like this. And so generally what I do personally is I use pre trained models to create baselines. So this gives you a kind of good starting point. And then once you understand a little bit about okay, what the limitation is on the domain, then train your own. But yeah, I think personally the rule has been to train your own embeddings yeah, okay.

Okay. That's a very philosophical question.

I can go and answer this one. So, so I think the NLP specialization does give you a strong base to be able to work on your own models and implement these algorithms in your own products. It also gives you a very good foundation to be able to read a research paper and understand it and just understand how these models work, how you can download them, how you can get them deployed in your own, in your own, at least how you can train them to, to, for your own products. So I, I do believe, like, it does give a very strong foundation and a strong base because by the end of the four courses, you actually implement 16 projects. In a way, each, each week has its own assignments, which in of itself is, is a little project. So I think after the NLP specialization, like if, if you work on a project that's, or contribute to open source, then it's going to give you a very good profile for recruiting and for getting a job after. And also, definitely it helps with interviews. So lots of the NLP questions you get in interviews, if you took the NLP specialization, you really understood each topic, you should be able to answer those interview questions.

Very cool.

Okay, I think we're here.

Text to image technology. Is it feasible to build the AI product that will transform text to cartoon images or meme images based on the current, advanced, current level of technology? I think OpenAI is trying to build one, but it seems they haven't published any products yet.

Yeah, that's right. OpenAI have a model called Dall E, which basically you can provide some text like a kangaroo on the beach, and it will then take that text and generate an image of a kangaroo on the beach with some accuracy. It's true that they haven't open sourced the model, but there are actually two community efforts underway right now to do this as well themselves. So there's a Dall? E mini project on the Hugging Face Hub. So if you go to the Hugging Face Hub and you look for Dall E Mini, you'll find the work in progress there. And it already generates quite nice images. And just today on Twitter, which is where I get all my news, there was a release by a Russian company of a Russian Dali. So here you can type things in Russian and it will generate actually pretty good images. So I would say the technology is like, still kind of cutting edge. Like it takes a huge amount of computing power and very large data sets to make this work, but it's probably going to become commoditized in the next few Years.

Future of Transformers. What is. What future do you see for Transformers in general and at hugging face, besides tweaking the size and structure? Like, what breakthrough architecture or training strategy can help Transformers going ahead?

Do you want to take that, Eunice?

Yeah, I can try to take. So I think, like, Transformers, maybe they're going to speed up, like, processing powers for training these models.

They'll probably in the size and structure. Like, what breakthrough architecture or training strategy can help Transformers going ahead?

Do you want to take that, Eunice?

Yeah, I can try to take it. So I think, like, Transformers, maybe they're going to speed up, like processing powers for training these models, they'll probably try to make them more efficient, maybe by speeding up, I don't know, look up at their attention phase or by doing some type of reversible network inside, so I can see the transformer network maybe becoming much smaller and keeping the same amount of power in capi. To make them more efficient, maybe by speeding up, I don't know, look up at their attention phase or by doing some type of reversible network inside, so I can see the transformer network maybe becoming much smaller and keeping the same amount of power in capturing dependencies in very long sequences. And then, yeah, other than that, I trained dependencies in very long sequences, and then, yeah, other than that, I think, at least so far, they're probably one of the best architectures out there. I don't know if, Louis, you want to add something else?

Yeah, the only thing I would add to that is that what I've seen that's quite exciting is that Transformers started in nlp, and in fact, they started in translation, and then they kind of quickly, say, took over nlp. But now in the last, like, two years, you can see them branching out into other domains. So there's now Transformers for computer vision, there's Transformers for speech tasks, there's Transformers for reinforcement learning, There's Transformers for biological processes. And so my kind of expectation is that we'll see Transformers become more like a kind of general tool across many different domains. And the thing that I'm really excited about is, like, the integration of these additional modalities. So the combination of, for example, text with audio with vision. I mean, if you combine these things, you start getting kind of close to building, like, really intelligent systems. And I think Transformers seem to be the tool that will help us get there.

Well said. I'm excited to see what ends up happening. I think we have a few more questions that have come up. Sorry, let me refresh and see if we have any other questions that have been uploaded since then. Okay, meaning slash unstructured data. Autonomy had this messaging around extracting meaning out of unstructured data. If you're not familiar with idle, they claim they were able to do this without structuring the unstructured data through clustering and few other techniques. Do you feel we're able to teach NLP to figure out things like the dog walked in the room, it was furry. Like, teach NLP to understand that the room is not furry, but the dog. Like, I feel like they're talking about the context here.

Oh yeah. This is actually a very, very hard topic. It's. It deals with coref and I remember actually one thing like it was fareed dealing with the doc. So I was actually trying. We were trying to build an assignment on core on the NLP specialization and I think I ended up falling on the Winograd schema. And it took me like two or three days and I was trying to make some progress for the programming assignments and then I realized, well, this is kind of an unsafe.