DeepLearning.AI NLP Learner Community ft.Rishit Dholakia
Rishit Dholakia, an incoming NYU MS student, walks through his NLP research and industry work: a demand-forecasting paper using word embeddings and clustering, an Auto Suggest tool that generates product descriptions with transformers, and an Auto Writer encoder-decoder that builds descriptions from product attributes. He covers practical lessons on data cleaning, deployment, and software engineering.
Chapters
Hello everyone. So welcome to the DeepLearning AI online mini series event for NLP learners community. I am really honored and excited in order to present my experience and be a guest speaker. Today I'll be speaking for around 15 to 20 minutes about my experience and my journey in artificial intelligence. Then I'll be answering all of your questions on the Slack group that's there. So don't forget to post and comment all your questions on it. And before I start, good morning, good evening, good afternoon to all of you learners from wherever you all are. So let's get started. Just start with my presentation. Okay. So welcome again and I would like to thank the Deep Learning community for giving me this opportunity to actually present my experience and even at the same time. Thank you Alice a lot for making and arranging this event so smoothly. And so today's agenda will be about like I'll be talking about myself a bit about my experience. What have I done before about my research publication? So that was with respect to nlp, that actually got me more into the field of natural language processing. Then we have Auto Suggest, which was a company research and development project that I'd taken up over there, the auto writer as well and how to keep learning, some suggestions on how to keep updated with the latest trends in natural language processing, AI and then the question answering session. So about me so I'm Rishi Dhalakia from Mumbai, India and I've completed my Bachelor's in Technology and Computer Science from National Institute of Technology Surat. I've been an analyst from 2019 July to 2020 July at OGAM, a Merkle company where I worked on many products and was a part of the Jarvis team, which was the cognitive computing team of the company. And then now I'm currently alpha tester at Deep Learning AI and my forward journey would be that I'll be doing my Master's in Computer science and specializing in natural language processing and machine learning at New York University. So yeah, let's start with my first publication. So it's so this was actually a cognitive demand forecasting project, you could say in my final year, which was then published as a research paper. Over here we actually tried to tackle a problem of under stocking and overstocking where we tried to accurately determine and predict what would be the amount of stock that would be required at a particular day and a particular time of the day for say normal groceries and say perishable as well as non perishable goods. But what we tried to do and we came up with the research Was how could we include NLP and other novel features and see how it would trade off with the traditional approach of just cleaning the data, pre processing it, feature engineering it, and then passing it on to give predictions. What happened was that we actually thought that what if we had a product and we are able to get the meaning of the product and give it a categorical feature to it. So we wanted to see how the feature would work. So we use NLP in order to categorize the items, the products, in order to improve our predictions. And we even included the novel features like sessions of the day. So how do we do the NLP inclusion? So this was one of our research topics. So we use a combination of word embeddings and unsupervised learning. So word embeddings, as you all know, would give us the meaning of the word. So we had products like say T shirt, table, you know, cups, towels and all these things. And what happened was you would get the meaning of this word so that, you know, all items like towels and cups and all that, they would all get clustered into one type of group. And that would be done using unsupervised learning. So that's how we went across. So the subcategory input was technically a normal input that we would given our words and then we'd get the word embedding vectors. Because, you know, as you all know, that computers, they understand numerical more than, you know, just normal words as we tell. And then we would reduce the dimensions. So we would take around just inserting all 300 dimensions. We would take around like five dimensions to six dimensions as a prerequisite. We actually used a pretend word embedding model itself and word embeddings, which were the Google's Google News word embeddings. And then we, you, we, what we did was we use the clustering, the K means clustering method in order to cluster these items. So this was actually through a search method where we actually had to try different permutations and combinations of the dimensions and clusters, get a perfect amount of the perfect cluster value and the perfect dimensions in order to categorize them in a much condensed way, I could say. So after we did that, we needed some sort of a benchmark since this was not done before. What we thought was we take a practical application of it by actually going onto the web, going to some of the websites and actually seeing if they were actually present in these, if these items are present in the specific categories or not. So as you can see on the right, this was the generated data that we had. So like we had sponge, as you can see, it belongs to a Category 5. Then napkins, cups, plates. So it was, I couldn't say it is. It was 100% categorizing the item, but it was to a good amount categorizing the item. So once we did that, we needed to now go further and check how well it helped us improve our predictions. So this was a data that we finally passed in after a lot of engineering that we did. So you can check it out in the paper. I'll send the link later on to all of you. And this was the cluster value, as you can see that we added as an extra. And this was the model that. So initially what we did was we trained our model through artificial neural networks and then gradient boosting algorithms like XGBOOST and lgbm. As you all can see, that XGBOOST did perform really well for us on the train set and the test set that we had. And I would say the train set and tested could be around like 80, 20, 70, 30 as well. And once we did that, now we wanted to check on the XGBoost, how would it have performed without including the novel categories that was a part of our research, without including the sessions of the day as well, which was a part of our research. And we could see that over here there has been a lot of, I could say the test that had not performed the model not performed really well on the test set. And it performed really well when we included both these things, the categories and the sessions of the day. So this was some research part that we did. And this is what gave me a bit of an interest into, you know, exploring NLP further on for, you know, such normal for such traditional tasks that would have. And now moving on to some of the exciting thing that was the project that I work on, which is auto suggest. So what would happen usually in the company is that the retailers, they would have requirements of writing contents. And we had a content writers who actually help, you know, make all those amazing descriptions that are there for retailers. Like, you know, just give an example of many of the companies that are there. Like, you know, you know, Amazon's there, then you have Flipkart and then you have Best Buy. So you see the descriptions and they are pretty well written. So in order to help all our content writers, you know, instead of going in, always researching and seeing what is the type of the description that needs to be written every time, and then started writing it, what is that? What we Thought was let us provide a suggestion tool to them just like our Gmail compose that is there. So as, I mean keep on writing. You would end up producing some amount. You generate some amount of recommendation like five or six recommendations for them. So they could select any of them and then continue further and get an essence of knowing. Okay, this is how we would have to write it. And this would be like modeled separately for different retailers as such and sometimes even for different product categories. So this is how we went on with the data pipeline. First we had a level and a hierarchy so category and subcategory which we had to decide. And this was one of the most important steps because we needed to know at what level should we start in order to know how much the model learns. So it's. I'll give an example later on. And then we had data collection, data cleaning. That was a. Data cleaning was one of the most important methods. I'll come across it and tell you about it later on. And then we had one dometic generation after which we did a model training and then deployed it to our Jarvis platform. So as you can see it talked about a hierarchy. So we thought either do we train the model and apparel level with apparel level which would be like you consider data from like you know, the woman's category, which means all the data that is there in order for us to train the model and as well as the data from the men's concatenated combined data or should we make separate models such that we only train a model for women on retain a model for men. So like this is just an example I'm giving. So there were different types of categorical levels that we had to first sit and analyze just as a benchmark for us to start. So we start and we would say like I would say it would be good to start from a top to bottom approach. So that'd be a much better way of doing it. And then what we did was we did was a collection of data that was there and this was actually done using our database that was there. And since we had a database, all the data was present with us from the data so we didn't have to externally scrap data or do anything with it. And then you know, using SQL queries and all that you can retrieve the data. Then we had data cleaning. And once we did the data cleaning this day was like one example sentence that we use in order to pass it on into a transformer based model for us to generate the recommendations. This is just a flow diagram. I'll be Going a bit into detail. So with data cleaning, so the to all of you is don't expect your data to be as clean as you get structured Data like the IMDb data sets and all that, sometimes the data can get really messy. So in industry level you will have to expect sometimes really messy data and have to work a lot on the data cleaning. So some of the data cleaning that we worked on was like removing the HTML tags that are there, then deduplication. So deduplication. I would talk in terms of say you had two Apple MacBook Pro models, okay, they had just different configurations but the same content. So even that data was provided to us. So for us we had just removed one of the data and just include the unique descriptions that were present with us. Then we did was masking. And like this is where I would suggest everyone to think from a model perspective. Because when you have a data in a format like, you know, as you can see a new Surface laptop with a series number and you know, even the numerical number, what would happen is you wouldn't want the model to learn all these series numbers as such, but you would want the model to learn these placeholders that, okay, after I type say a model name, I would want a series to be there and maybe after the fuel I would want there to be a value as such. So what would happen in this case is that we would apply a masking method in order for us to get this data and in this format. Then the next thing that we did was word embeddings. So with respect to word embeddings, what I would like to say is that the reason we did word embeddings is so that we could provide a standardized word vector format, word vectors for the entire company itself. So anyone who is working on a diff in a different team for sentiment analysis and they require a retail related, you know, word embedding, it could be standardized across all the different projects that we're using. So this was something we worked on and we provided. So we tried different methods of word to wake Word to vec fast text and then there was Starspace. So Star Space was E commerce word to work word embedding models. And many people have been using right now and we use perplexity as a measure to evaluate which embedding model work really good for us. And then we come on to the model training. So I know that Gmail compose and all that are usually like you know, one lines of data you might train and measure suggest a part of that line. But here, since there are descriptions of say three to four lines. How would you do that? And the other point was that we know that LSTMs and RNNs were not that great at giving long term, you know, output content as such and giving long term description that actually made a lot of sense and capturing the context as such of say a previous person. If you guys would have done the sequence model courses or you know, even course three of the NLP specialization where I noticed but still what we did was we tried out just as a benchmark on stacking up LSTM units and seeing how it worked. And to be honest it didn't work that great. We had to work on further. We saw that the loss wasn't decreasing, there was no learning and you would get an output as like that. And then we tried to transform based model, then we tried transform based models and those actually worked well for long range sequences. So yeah, we went on and we worked on furthermore with different transform based models and all that. And the metric to evaluate it was the BLEU score. So this is just an overview that I could give with respect to the project. I worked on one of them and then the other one was auto writer. So this was another exciting project that we worked on and it was, it was actually one of the most challenging as well. So what we had to do here was we try to simplify the work of the content writers. So instead of them continuously every time typing for the retailer, we would follow a retailer that they would have, they would just need to input say like the T shirt, the brand, the category to which it belongs to, it would automatically generate the entire content for them. So I know this was like the real AI application that I would say that we try to work on. And so this was based on the concept of an encoder decoder architecture. So this is how the data was. So we provided T shirts. Then we, so we provided an example. Like the attribute name would be T shirt, it would be men's, it would belonging to clothing, it will belonging to fashion and the brand and pass it into an encoder and at the decoder you would end up getting this entire content that was there. And to tell you, the encoder decoder were transform based models as well. So this is how it worked. And the most important part of this was the data cleaning. As I can say, the data cleaning and collection was a bit different compared to the auto suggest as the data collection was from a different set and a different system that we had to fetch. And the data cleaning was as well, different because the description and the products had to be taken care of at the same time. With respect to data cleaning, what we did was that our data contained so much of unwanted sentences like say only or kids above the age well or say beware of this plastic, the safety precaution sentences. So we try to remove those sentences. And then there were other sentences and unwanted attributes. Attributes in a sense in the description as well. So we had like the sizes available were small, medium exercise. I told you all that the data can get pretty messy. So these were some of the important cleaning that we did. There was a lot more of cleaning like punctuations and all that that we took into consideration. And the other important part was the product name. So the product name would technically be like, you know, the T shirt or whatever products we are going to specify. So thinking again from the model perspective, what I would say was that if you had a product name, say like never give up T shirt, the logo name was there as well as the T shirt, the never give up would not make any sense to the model. It would rather end up confusing the model more. So instead what we had to do was we just had to pass in T shirt. So how would we have done that? We used POS tagging method in order to solve this problem. So you know, you would just get the noun values and you just use them concatenate with the rest of the attributes. And for us the attributes are provided to us from before. So we had different columns of attributes that actually help us, you know, concat all of them and send them out into the encoder. And just for information, usually the attributes are stored in a pim, which is the product Information management system. Just as a general knowledge question for general knowledge thing to keep in mind for technical tasks like these. So we worked on the model generation now and for input, we didn't just provide an input as a bag of words straight away like say T shirt brand. We provided the input like this is a T shirt. This is the name of the attribute belonging to which category and belonging to which subcategory like that. So we provided like an entire sentence that get encoded from the encoder and then you get the decoded output. So for encoder decoder as usual, we started off with an LSTM based model. Technically there was no learning at all. It didn't learn anything no matter how much time we spent and all that. Then we worked on with transform based models and had had a really good learning and some really high blue score. Rates. So this is still an R and D project that's going on I guess in the company. I'm not so sure right now. And that's what. So this is my industrial experience as such that I had on in the field of nlp. So some of the lessons I learned Regex is really important. All your pre processing and everything that you require a lot of regex. So always make sure when you're doing NLP tasks, you know your regex really well to work really well with all the natural language text that we have crawling and scraping. So if you guys are doing a personal projects, you guys might not get data ready in hand, you might have a different idea and you guys might want to work on with that. So for that I recommend all of you learn crawling and web scraping and everything and then analysis of the categories needed and efficiency. So what happens is your practical environment on many deploy a model or something and when you're training it are two different environments. So you need to keep in mind about how, how to go on with which category and how at what level is the context actually being learned. It should not be that you pass in data with respect to a baby T shirt and you get an output with respect to a T shirt. But that gives a lot of context with respect to adults like men or women like that. Right? I mean I'm just not gender biasing but I'm just thinking in terms of the category to which it belongs and the one goal and interact with your domain experts. So like we had a lot of interaction with the content writers. Like every model that we produce, every model that we build and the content we generate, we'd actually go and interact them like are you fine with this type of model? Are you fine with how this model suggests you something and all that. And one more thing I forgot to mention in case auto suggest we went on was a bigram and trigram method. So every time you type in three words, the content writers would type in three words, you would generate the rest of the three words and the six words would again then pass on to the model and generate the rest of the three words. And that analysis was done as well. So. So you didn't just pass a. You wouldn't just tell the content. I just pass the word the and it generates. You need to pass and see the product name as such initially the first rewards and then it works on and then the garbage in, garbage out rule. This is like the most global rule for any ML engineering or any ML work. You do a model, you create so the garbage that you pass in and the garbage, the garbage that you pass in will be the garbage that you get out. So if you pass in a very dirty data, no matter how much you train your model, you're not going to get the best output data that you want. So always make sure you spend enough time cleaning your data, understanding your own data and making sure that the way you understand your data is something that even the model understands the data. So that's how I would say that everyone please make sure of your cleaning, cleaning the data because that's really, really important as well as the distribution of the data. Because modeling is usually less taxing task. I could say then the latency of the model. Latency of the model I would say would be in terms of how fast you can get the output for something that generated. Sometimes what happened is you might type three words. It might take around three to four seconds to just give you the recommendations. A lot of time you need to look into latency as well. So of the model because maybe generating the model it performs really well. But if it's not fast enough, it's not efficient enough, it's not going to be of any point to anyone. They might, the users might end up getting frustrated. So there's a lot of software engineering techniques as well. I recommend people to know dynamic programming because that was something you will be used, you would use later on in order to, you know, increase the efficiency and all that and how to keep learning. So with respect to the learning part, I would say that the field is growing exponentially right now in terms of natural language processing. There was a time for computer vision and right now there's even the time for generative models and all that. So the best places that I would suggest is like you know, the batch by deep learning AI where they publish weekly blogs on the different innovations that's going on in the industry so that you guys can get an idea on where the algorithms and industry has reached to in terms of AI. Then you have medium blogs, then your MI Technology Review. So MI Technology Review is a bit more technical and it gives you a good understanding on different in depth technological advancements that are going on. Then on YouTube you have Lex. His videos are pretty good with all many of the top AI researchers and industrialists as well. And two Minutes paper is technically all the latest paper that have been published. How do you know just grasp them when you're just going before to work or you just come back and you're not in the mood of reading an entire paper. Just getting a gist of it and then reading it when your time to see if you're interested or not. So Two Minute Paper is one of the other YouTube channels you guys should subscribe to. Then Paper with Code is another good site website I would say because for persons planning on becoming a researcher they would require to have that skill of you know, actually being able to implement the papers. Which means the way we are explained RNNs is not the way it's written in the research paper. It's much more mathematical and much more technical. So you guys need to understand that notations and those the wordings and the language of writing a research paper as such. And how do you translate them into code. So Papers with Code is one of an amazing website where they have the paper as well as the code given side by side and then at the industry level on how they have deployed and some of the interesting things like company blogs, we have Google AI blog. So they publish a lot of the research that they have been doing in Google. There's Facebook AI as well, there's Airbnb and just couple of days back I'd seen about Instagram. So how their explore actually works, how they use machine learning to actually give you the recommended explore options, different recommendations. So that has been one of the best. You know these are one of the best sources that I look through and go through often in order to get a good gist of the different developments going on and projects. You keep on learning through projects again and again I'm telling, I would say that it'd be better to take up a project do it. Suppose if someone has already done the project also it's a good idea to just do it so you understand the entire pipeline on how people actually do some sort of an end to end machine learning project. And you all could go on with that and suppose you come across a different idea, you all could learn even some back end deployment of flask and try out different methods and just go crazy with your ideas. So that's my recommendation. And now it's time for the question and answering session. Just seeing on any questions and answers anyone else? Okay, so we have a question from Vig on how close is the industry to automatically classifying and generating intents? So as long as I worked the knowledge that I have is classifying intents is pretty much done right now automatically. There are many different libraries that are available like Rasa and all that many companies use right now. So in terms of say there are, there are places I guess and even in Our company that we had utilized intent, like there are retailers that might require such type of, you know, intent classification and all that. And it's usually done using these algorithms that are there. And I could say that much about it. Like a lot of other companies, a lot of startups I would say are using this automated intent methods provided by say Google as a. So that's probably Google as a service. Some of them even make their own classifiers, intent classifiers. But even with respect to what I had seen like in my internship that I had done with respect to deep learning, one of my colleagues in 2017 had worked on intent classification and the advancement is much more right now, which means there are more resources available, I would say. So it has got quite close. But you cannot say that everyone's completely adapting to that. But yeah, it's still a field where you can work further on different types of intents, the extent of which you're using a natural language to generate what type of interns, the diversity of intents, all that. Then there's a question by Rajan who says that I need help implementing Siamese learning for negative mining in Pytorch to come. Months still stuck in the problem. Anybody help? So with respect to this, I might not be the best person to answer be able to answer this question, but I would need to be able to look at your code and maybe we could discuss on this a bit later on, you know, maybe have a discussion, see your notebook and see where things are going wrong maybe. And I guess we can work on that later on. So next question is by Sonali. I am looking forward to build a chatbot like Haptics Conversational AI would love you to guide me through the correct direction to be able to do so. So for those who don't know Haptic, Haptic was one of the chatbots that JIO had acquired, one of the chatbot companies. So Sonali, with respect to building a chatbot, I look at it in two ways. So do you if you would want to learn on how to build a chatbot in the terms that say, for example, you want to know all the way from your intent classification, you want to develop that sort of a model and then go on to even developing a different model to make a dialogue to generate all the different responses and everything, for that it would be better to go through sent Deck's YouTube channel where they actually deploy it from an end to end method, which means you go all the way from collecting data. After you collect your data, you actually go through and clean the Data and then you make different intent classifiers, you train the model for intent and do all that entire process and make the chatbot even, you know, write different scripts for all the deployment. So that is one way of learning it. And the other way I would see is using pre built frameworks that we already have. So there's like Rasa framework that's there. So what happens is all the intent, classification all is abstracted with on there. So you will be able to make a haptic, conversational AI customized model like that. You wouldn't find a problem as such in making a model you know, which is like an end to end and gives you responses on the way you would want your startup or your company to or your personal project to work upon. So I don't think so that would be a problem. It just depends on how much of a learning you want. So if you are, if you are moving at a faster pace on you on just making models of, you know, just deploying it and getting it out as a devil development, then I would suggest go using Rasa and if you would want to go on learning it from scratch, then please do visit some of the on medium also there are blogs that actually do it that actually show you how to implement these models from scratch. There was one project that I also worked on which was like a stack overflow assistant and where we actually modeled the entire chatbot from end to end. So it collected the data, we created embeddings, we had a chatterbot, we hosted it on Telegram and so yeah, so if you want you could connect on later and I'll try to drop in links for you know, resources in order to make such chatbots then. Sorry, there was one personal message that I received from Shivang Sharma. I need your advice in one section regarding the specialization. I have not completed it but I am still confused about the real world projects in nlp. Okay, do we use tracks in real world nowadays? Because I am familiar with TensorFlow and is this how we implemented the project in real world as we have completed in this specialization? Can you please help in getting where to look more projects for nlp? Oh yeah. So I guess this might help all of you a lot. I'm not sure. So the first thing is about real world projects. Some of the examples that I've showed you is actually a career world project that we use like in terms of you would use NLP1 in terms of attributions. So many of the retailers, what they would want is they would want to attribute the data so you use POS tagging and all these natural language processing techniques to do it. Then the other NLP related real world projects that would be are, you know, sometimes what would happen is people might not want to read the entire news. So there might be, there might come over time when people would just, you know, just scan the particular news section and they might get a summarized NLP related, you know, summarized text to just, you know, know what has been going on in the entire passage. So something like that, you know. So these are some of the real world projects that you can implement. One of the personal projects that I had implemented was a drumbeat generation. So like this happened when I had a conversation with my friend and he told that he was just a guitarist and he didn't have a drummer. So I thought what if we could actually, you know, have some sort of a method where there are drummers and guitarists, they have a recording, they have some sort of a data and we are actually able to, you know, for generate drum beats for those particular guitar notes that are being played. I though I was not that great at the domain. I would, I'm still working on it and I'll be. And I have been interacting with my friend lot with respect to this since he has a lot of knowledge on all the musical instruments and all that. So there are some of the real world projects and do we use tracks in real world? So as I would say that tracks is currently new now it all depends on the world application. So like if you go to any startup or any other place currently they're using TensorFlow may not be well versed with tracks because you and I come across tracks from the specialization itself. So what I would say is that learn tracks, it's good to have a knowledge of different, you know, frameworks because it'll help you later on in, you know, getting across different platforms and knowing the different ways on implementing things. Maybe later on when tracks becomes much more popular like TensorFlow or PyTorch. So it's always a good knowledge to have tracks. And is this how we implement projects in real world? At the core I would say yes. But there's a lot of software designing and software engineering that goes in. So you would make it much more organized in the sense that in the notebooks you would have seen there's a data cleaning and everything done in just one note, one notebook of assignment. But later on what you would do is you would have separate, separate Python files for your data cleaning your separate Python files in order to, you know, take in your data and you Use a lot of, I would say Python libraries and all that in order to get some sort of data automatically. So yeah, at the core you would be using these ML models, you'd be using the same concept, but there is much more to this. So like once you have the data, how do you send out the data? A lot of back end and a lot of software engineering also that goes in a lot of day script organization, I would say it that would be required a lot of unit tests that would be made. So once your model is trained how it works on the unit tests and all those things. So yeah, so you would have to learn a lot about different software engineering skills in order to implement it with efficiency as well. So like you know, if you're doing a search, you wouldn't just do a normal binary search that's there, I mean, sorry, a normal brute force search, you would end up using a binary search or you would use end up using different types of sorting techniques or something later on if you're working on projects. And could you help me get some real world scenarios? So this was a problem that even I had faced at one point. So what happened over here was that even I was not able to get across some new ideas. So what I thought was try reading more. The more you read the more you'll be able to understand different implementations and different ideas on where MLS being used as such. So what happens is I'd come across one of the Hummingbird. I'm sorry, not Hummingbird, the Hummingbird tune modeling by Google. And what would happen in that is that was one of the projects that a friend of mine had also told her what if you could actually work on something like this? So once you read about different projects like this, you might be able to take the same architecture but be able to provide something further on in that particular sphere. You know, I would say read different blogs, read on the latest developments that's going on because that's how you'll be able to know about the different ideas. You might. Different ideas will just track to you as in when you're working on some sort of one year reading on something, you're like okay, wait, let's. We could just change this a bit. We would be able to what if we are able to actually apply this particular idea with another detection algorithm or something like that. So I would say read a lot, gain a lot of knowledge on that because that's how you'll be able to come up with different ideas. It's not just going to come out of the blue. Moon at one point without having the knowledge of what are the existing platforms and what are the existing techniques and different deployment and industrial applications of what is being used currently and come back again. There's another question by Sachin. What incidents led you to focus in NLP with a huge domain of machine learning? So actually this happened, it's a story back then in my college days when a friend of mine actually he had generated Shakespeare text and I was new to all this that time and I was really intrigued by all this and I'm like wait, how is it possible that there's a system that actually stores in a memory and actually utilizes the memory in order to generate new sequences? I felt like that was like the real AI. And that's when I went on further, you know, analyzing different first was deep learning models like recurrent neural networks, LSTMs. Then I went into the field of NLP as such knowing about stemming lemmatization and all that. And then in our college we had a five day seminar just on natural language processing and that's how I got into the field of nlp. And once I went into the field of NLP I got opportunities to work in a startup where I had worked on like speech to text machine translation that helped me develop stronger skills in the field of NLP and that was the time in Transformers and all that also had come into the limelight and I was. And those all things were quite intriguing and that's how I got into this field of NLP that was there. Then there's a question from ibona if I'm not wrong, I have a few questions. If it happens that the observation A goes to the training goes to the training and a dash goes to the validation, for example reverse complement of DNA or otherwise transformed within augmentation, should it be considered an undesirable action? It does not go into that. It does not go into the test set which is completely excluded from the training. But should it still be split in such a way that the original sentence and its augmentation always go into one set? I will be very grateful for your comments. According to me, I would suggest that you would need to pass even the complement of it into the training set. That is one option because at the end of it maybe your RNNs or LSTM models they learn on sequences so if you reorder them or if you have have a different sequence, it would make a difference. Or you could do one more thing is like just a normal if condition that if you have a complement of one of the strings then it would be better to, you know, just change the would be able to detect and search if there's a complement of it, reverse it back and then just, you know, ignore and just use the unique data that's there. So I would suggest that it should be a part of the training or the. Or the. I would suggest you should be a part of the training more than the validation because even if you look in image I would not say image augmentation but like even over there so you just change the image in such a manner and you provide as a part of the training rather than just, you know, as a validation. So I think providing it as a part of the training would be a much better option. And if we want to somehow verify the quality of the model with the genius during training example word to back then as I understand it, we should observe the model dot get latest training loss. But there is information that this is not the best choice. Okay, so what are the measures if there is only a check by using the classification. So actually this is one of really good questions because I had come across the same problem when I was making my word to vec models for and you know, making all those different embeddings for generating different embeddings for the text. So I had spent a lot of time trying to figure out which is the best method in order for me to go about testing which of the model which is which model would be better for my generation. So I would say that what I came across was the perplexity score. So the language model models would work best on the perplexity score. Till now I've been using that and it did work kind of fine with that. And the other method I would say is manually. Just I do make cosine similarities between different words, you know, and see if they're actually related or are there are there any discrepancies as such in that I know that can go it that could be a bit time consuming. But you could always write an algorithm in order to, you know, take random words, random pairs of words that you might have and see how it works such but perplexity is one of the best language model metrics that I would say you would use in order to check the quality of your model in word to vec fastech. Should the number of epochs be related to the size of the body? Do you know of any study comparing obtained embedding according to the number of epochs? I don't know any study as such, but when I was working on it, I did find that increasing the number of epochs did make a difference. But on the content, at one point I might be even. I may not be so sure of it, but the dimensions made a lot of a difference. So the hidden dimensions of the word embeddings that. The dimensions of the embeddings that did make a lot of a difference with respect to your, you know, in comparison to your data. That would be the size of the data. So yeah, this is what I had. I may not be the best to give an answer to that, but yeah, this was through my experience. What are some of the challenges or interesting problems you worked in the field of NLP in research as well as job. So what I would say is that currently it was the AI, the auto writer. That was one of the best experiences that I had in order for me to, you know, know a lot about word embeddings, knowing when to use it when not. Sometimes actually word embeddings do not help out a lot. Like for example, the question answering system. I read in one of the blogs as well that in that the word embeddings didn't help that much. And then I learned a lot and it was really challenging that part of nlp, the AI writer. Then my research that I worked on was as well a lot challenging because we had, we didn't know a proper method to come up with something, you know, different, as I would say. I mean, we were just limited with the data and we had to try to find out a way to come across.
