Wolfram Student Podcast Episode 9: Predicted Native Language of a Writer Based on a Sentence

Hello, everyone, and welcome to the ninth episode of the Wolfram Student Podcast. I'm Sam.

And I'm Rashank.

And we're your host.

This is the Wolfram Student Podcast, where every fortnight we dive into a new innovative project done by high schoolers using the Wolfram language.

For our ninth episode, let's welcome Muskan Mohajan, who will be discussing her project on predicting a writer's native language given a sentence. Hello, Muskaan.

Hello, world. Hello, world. From. And thank you so much, Sam and Rishang for having me. I'm Muskan.

Yeah, no problem. It's great to have you.

Could you give us a brief sort of intro about your background? What's your name, grade, country, etc?

Sure. Deal. So, I'm Muskan. I'm a high school graduate from India, and to make the most out of my gap year, I participated in the Wolfram High school summer camp 2022, wherein I think I was groomed and I was allotted a project very aligned with my interests in linguistic, natural language processing and education. As mentioned by Sam, I created a native language identifier for English texts and I am extremely excited to dive more deeply into it and tell you all about it.

Cool. Can you tell us a little bit about camp? Because I think you went to 2022 camp, right? How was it?

Yes, for sure. So I had my earlier apprehensions about the Wolfram High School Summer Camp in the online format because I was not sure if you could get a lot out of that format. Right. However, attending the camp proved me wrong. I think it was a really enriching experience of two to three weeks wherein I think we were provided with just the right balance of work and play. We had extremely amazing workshop organizers who organized lectures on topics like data visualization, interpretation, and my favorite topics like neural networks and natural language processing with the Wolfram language. Don't let these words intimidate you because other than being specialized, it was extremely beginner friendly with the first few days being dedicated to induction and helping us understand the Wolfram language way better and helping me fall in love with it at the same time. So I also mentioned play. We were divided into groups and yes, there were a fair share of icebreaker activities and just a whole lot of opportunities for us to express ourselves to our peers from, I believe, around eight to nine countries. So, yes, an amazing experience overall.

Great. Yeah, no, I definitely agree. Sam and I have both attended the ultrasound camp in the past and that definitely sums up our experience. So, yeah, just to kind of Break the ice a little bit. And before we dive into your project, do you mind sharing with us your favorite Wolfram function if you have one?

Oh, sure. So I mentioned natural language processing, right. So I am a phonetic when it comes to the field of question answering, giving the computer some context, giving it a question and hoping for an answer in return, wherein the computer just has to look through the context. And wouldn't you know that Wolfram has a function to perform just that, and it's called Textual Answers. I love playing with it. I love hiding my answers in a very subtle and cryptic format in the context that I give it. And I'm impressed every time Wolfram gives me the correct answer. So, yes, in a nutshell, textual answers is my favorite function.

Yeah, that sounds like a pretty amazing function. Cool. So before we start, can you just give us a quick general summary of what your project is about?

Sure. So my project, as the name suggests, is about classifying person's English. Classifying a person's native language based on English text that is inputted by them. We do do this based on the assumption that the way a person who is learning English as their second language shows some kind of general trends in their writing patterns, the kind of grammatical errors they may make while production, and it is largely dependent on their first language, which we call the native language. So yes, in this project we aim to explore deeply that.

Yeah, so I mean, I took a look at your notebook beforehand and it sounds like a very interesting project, honestly. And yeah, I wasn't really sure that you could like, when I looked at the title, I was like, can you actually predict, you know, someone's native language reliably if you look at them right in English? So yeah, I'm excited for you to share with us how you did this and you know, how you used logic at each step to accomplish the task for sure.

So as I stated earlier in the summary that we are using the assumption that an author's native language, which is their first language, it will somehow influence the way that they produce text and speech in their second language in this context, the second language being English. So what we are trying to basically achieve in this project is to find certain features that we can use to find out native language specific patterns in the second language text, one that can help us trace it back to an author's native language. And while I'm mentioning what this project aims to do, I haven't really touched upon why we need to find a person's native language. Does it have any real world applications? Yes, it does. Native language identification is a really popular task when it comes to natural language processing, and it's a raging field when it comes to forensic linguistics. However, additionally, and something that deeply resonates with me is that it has a lot of scope in education and second language acquisition. Let's take the example of universities where you have got people coming from diverse nations and diverse backgrounds trying to study English, because English requirements are a must for almost any university abroad. So I believe that having such a text, which can help the university instructors predict a student's native language based on text written by them, could help them adopt maybe native language specific strategies to help their students enhance their English language proficiency. And yes, that's what made this project so interesting for me. So now that I'm done with the motivation, let's ask ourselves the question which may have been in our minds for pretty long, and that is, how do we make a machine distinguish between a Hindi learner's writing style and a Chinese learner's writing style? Because clearly, I think that's something that we can't infer from our naked eyes, but something that a computer can if it is fed with data of English texts written by students, of diverse students speaking diverse first languages. And for that, we turn to various linguistic features. Now, I will take you through some of the linguistic features which we think may have some form of merit in helping us predict a person's native language. And that is the kind of lexical feature breakdown in a person's text written in English. Now, this could include the word engrams, character engrams, lemma engrams, and lexical richness. To break down more on these features, I aim to find out that is there some kind of pattern in the kind of words and the word orderings among the students who produce the English text that can help us trace it back to the first language? If not, can we find some patterns in maybe a more granular level, say character in grams? And if that's not the case as well, we thought of turning to a much more grammatical side and that is finding out parts of speech engrams that if we were to analyze the kinds of parts of speech that is used in the English sentences and the engrams, like the two to three word orderings of these parts of speech. Parts of speech being terms like nouns, determiners, adjectives, and whatnot, can it help us trace it back to the first language of a person? You can see that I'm not asserting that these features can help us. I am asking these questions because my project was based on investigating that. Other than that, I also thought of resorting to qualitative features like named entities. Is it possible that a Chinese, a Mandarin student who is trying to learn English would use certain named entities more than say a student who's a Thai student who is trying to produce the same text in English? And yes, with that information being there with us, we'll see that these are a lot of features. How do we pass these features into a computer? We do it by creating a list called a vector of around N features and trying to train a machine learning model. With that being stated, let's move on to the next part of the project and that is collecting the data, which I'll tell you is the real hassle. Don't let the world tell you that choosing the right machine learning models and coding is what will take up most of your time. No, it's collecting the relevant data, trying to make sure that it's within your budget and then eventually cleaning it up is the real task at hand. So when it comes to native language identification, since it's a very established task, TOEFL 11 is the benchmark data set for this problem. And yes, it sounds pretty fancy and let me tell you, it is. It is currently hosted on the Linguistic Data Consortium by the University of Pennsylvania and me and my peers and my ment at the Wolfram Summer Camp tried hard to navigate through that site. However, irrespective of the numerous mails and requests sent by us, we could not get the data set as it was locked behind a paywall of thousand dollars. So I had to say goodbye to it with a heavy heart of course. However, scouring through many websites and scouring through Google, we found out yet another corpus and that is the icnale and this basically consists of English texts written by students from 10 countries. Mind you, 10 countries in Asia and those countries are highlighted over here. Now this data set basically consisted of the English text written by grad students from these 10 countries on very homogeneous topics, I.e. smoking and part time jobs for college students. Now it was really convenient for us that we had. We didn't get randomized blocks of texts, however coherent ones, all written in the for the same context for the same topic as that would make a data analysis more easier for us. However, when I found out this data I was happy that I have something to work on. However, the question persisted that what I am getting is country names and essays. Country names, China, Hong Kong, Indonesia, being mapped to the essays written by the students from there. However, what was my initial task. It was classifying a person's native language. And the idea and the concept of native language is very debated around the linguistic community. How can you tell what's the native language of a given country? Countries like Singapore, they are so linguistically diverse with comparable portions of population speaking Mali, Tamil and Mandarin. How are you supposed to tell what's the native language of a given country? And that is something where I face some struggle with. However, I decided to shove that away by working on the assumption that the country's most spoken language be termed as their native language. And yes, I admit that might introduce some error and some variation in my results. Countries that were extremely linguistically diverse, like Singapore that I mentioned earlier, were removed from the analysis altogether. And therefore these, this is the kind of assumption that we worked with in the project with China having their native language as Chinese, Mandarin, Hong Kong, as Cantonese, so on and so forth. As you can already see, Singapore was excluded from our data set. Now if I were to dive deeper into the data importing, even though I have got the data over here in a very convenient, very compact cloud, as a cloud object, the process of unwrapping it, organizing it was not easy. We had to work with spaghetti looking XML files, trying to go deeper and deeper into the XML hierarchy, trying to fish out the relevant data. However, Wolfram with its XML helper functions was extremely helpful with that. And the reason I'm making this point about data extraction and data retrieval is so that the viewers can realize that this is somewhere some of the most difficulty lies data extraction and trying to arrange it in a way that is well organized and well interpretable by the third person. And this is also where I met with my favorite, my, my favorite data structure and that is association. Needless to say, it has helped me really well in arranging my data. So definitely again one of my favorite data structures when it comes to the Wolfram language or any language whatsoever. Call it dictionaries, hashmaps, whatnot. Now let's analyze the data. I realized that I didn't have the same number of essays for all of this native languages and I didn't want any class imbalance to be introduced in my data set. So through shuffling my data set I tried to, I tried to make it more uniform. And this is the result that I got for every native language I had 200 blocks of text. And I think that made studying the data a whole lot more reliable. So what you can see in this bar chart is the number of essays and in your X through across your X axis you could see the languages. So for each of the language we had 200 essays to study. This is the bar chart for the essays about smoking. And this is the bar chart for the essays about part time jobs. The data is finally balanced, like all good things should be. Well, we were discussing the data. Now let's go on to extract the linguistic features that I mentioned in the beginning of my essay. So let me quickly walk you through the kind of features I wanted to extract from these essays. And those are the word count of the essays, the sentence count of each passage, and this is followed by the motivation of why I decided to include these document level features. This is because while I was creating the project, I also went across the academic literature for native language identification. And I realized that adding certain statistical features like word count, word length and sentence length and the sentence count can help us in discerning which language students from which native language have an easier time writing more sorry frame, expressing more complicated ideas, more complicated concepts, and therefore adding such features may be valuable. Now that we're done with the document level features, I also decided to introduce other features like the average sentence length. This again would help us gauge how linguistically complex the sentences and the essays written by certain by the learners of English are. Other than that, I try to gauge the complexity of the sentences written by the test takers was through the lexical richness of the passages, which was gauged using the type token ratio. Now, the deal with the type token ratio is that it helps us discern how linguistically diverse a person's essay is. And that is by gauging the total number of different words written by them, the unique words written by them in mathematical terms, the set of the words written by them divided by the total number of words. And this could include repetition of words. The deal with type token ratio is helping us discern whether a person has a high degree of lexical lexical variation in their essays or not.

Just a quick question. So all these like features that you determine. So I guess just to summarize your plan a bit better, so you're going to analyze the data you got or like the data you collected from the Internet and see how these the patterns in these features in those in order to like come up with in like patterns that tend to happen. Am I understanding that correctly?

Yes, that would be correct. Trying to gauge the patterns that emerge with different languages.

All right, cool. Thank you. Thank you.

With that being sorted out, I decide after coding up the helper functions for all of these features, I decided to visualize them and then study them. So let's dive right into those visualizations. So first was trying to gauge the word count of each essays and mapping them to the native language of the text writer. And this is the result I got. The X axis shows us the native language of the person who has written the text, and the Y axis shows us the word count. And yes, we can observe that there was just slight variation. There weren't much trends that can be inferred from this bar graph. And upon going through the ICNA LE documentation, I realized that they had to write under a certain word limit and could not go above a specified and they had to go above a specified word threshold. And then again they were specified by a word limit. And therefore I found that there was not much value in using the word count as a feature, at least as far as my data set was concerned. So I decided to find out the patterns in the number of sentences in the essays that was written by language learners from the various countries. And well, I did come across a greater variance over here. Therefore, I found out that yes, maybe there is some value in using the number of sentences as a feature while creating a machine learning model. After that, I started to analyze yet another document feature, and that is the average sentence length. Again, some variance was observed. Now, when I came across word length, I realized that when it comes to working with words, be it in finding their lengths or working with engrams, unigrams, is that stop words would always be there either as your hurdle or as your friend. So when I decided to chart out the average word lengths of the essays written by the writers from various countries, I realized that there wasn't much of a trend to be seen. All right, so I'll move to the type token ratios and visualizing the results that I got from there. And what I realized over here was from the whisker plot, we can observe that the essays that were written by the Indonesian native learners, they were lexically richer, followed by the learners from Korean and Tagalog. And the data that the inferences that I got from this whisker plot, they corresponded to the inter language genetic proximity. I told you about how if a language lies in greater proximity to the English language, then the people who speak that language as the native language would have a easier time introducing more complicated ideas in English. And the whisker plot exactly corresponded with that assumption. Now let's move on to the feature of word frequency. When I decided to chart out the word clouds for the essays written by the Speakers from various countries, I realized that there wasn't much of a trend to be seen. However, when I decided to remove the stop words, I came across a very interesting observation and that was that for the Japanese speakers I could find I could find out the word Japan and other country specific named entities. And that made me realize that we should add a feature where and that made me realize that essay writers, they tend to cite their country and cultural specific entity terms in their essays. And taking a lesson from this inference, I realized that if we were to create a machine learning model for this in the future, then having a named entity recognition pipeline and mapping it to the writer's country could be especially valuable. Now I'll move on to the parts of speech. The inferences I realized from mapping out the parts of speech and that was that while we were trying to find out where we tried to create the word clouds for the parts of speech and a person's country, that we came across a term called foreign words. And that helped me realize that sometimes while creating producing English text, a language learner might switch to his or her native language, maybe trying to find a filler word that they couldn't find the English corresponding word for. So taking a lesson from this, maybe adding a language identification module to our classification pipeline may help as well. So now that I'm concluding my project, I analyze the essays written by Asian English as a second language learners on the topic of non smoking in restaurants and conducted a linguistic motivated analysis on them. After visualizing document features like the word count and sentence count, I realized that some features like the sentence count and average word length can be used for identifying the native language using machine learning, I also visualize the type token ratios, which means the complexity of the essays written by the language learners and realized that yes, there is some correspondence to the genetic proximity between English and a specific language with the kind of ease that the students have in producing English texts. And finally, after visualizing the word unigrams, I realized that there is some value in adding named entity recognition pipeline to the classification pipeline as well. So my further scope of study would include using svm, using support vector SVM based machine learning classifier to classify these essays and going over parts of speech of n grams of order greater than 1. I hope that you could derive some value from the project. Thank you.

Wow.

Yeah, thanks a lot. That was a very detailed explanation and thank you so much for sharing every step of the process with us. So I mean, first of all, this is an incredible project and I was, as I mentioned, I was quite surprised when I saw the title and I was like, can you really do that? But yeah, I can see, you know, the different techniques that you use to accomplish your task. So, you know, if Sam has any questions, you know, feel free to interject. But other than that, I would. I'm really interested in how you sort of see this, how you see similar techniques being applied in more practical settings or in future projects or. Yeah, something of that sort.

Yes, definitely. So I personally think that we can apply such techniques to future projects on style variations. Like, you have got so many projects on a person inputting an essay and us trying to predict maybe the author of that text and this kind of classifier could help us maybe predict the writers of a lot of undocumented texts. So I believe the techniques that I applied in my project could be used in writer identification, yet another application of linguistics, a linguistic forensics.

Yeah, that sounds really cool. And now that you've like finished this entire project, what would you say is your favorite part of this entire process?

My favorite part of the entire process. I, I think something that may have vexed you was how much I stressed on the data and in my personal opinion, unraveling the XML was my favorite part, as was going through academic literature or native language identification as I realized that there is so much more to this field and how much I want to deeply explore it further.

Yeah, that's definitely an important part and I'm excited that you get the opportunity to explore this further. Are you planning on using other languages to do this in the future or. I mean, I know the Bullfront language is very user friendly, so perhaps you might want to stick to that for now. But what do you think? What are your opinions?

Oh, yes, definitely. So I think that for visualize, for visualization and analysis purposes, nothing beats this beautiful symbolic language, Wolfram. However, I have had prior experience in implementing machine learning models with Python. So I stated at the end of my essay is trying to train maybe an SVM classifier on the parts of speech and grams, the word and grams, and the features that I found. So maybe I'll try to incorporate Python along with in this project.

Cool.

Well, yeah, that wraps up our questions. Thank you so much for being here and yeah, thank you so much for sharing your project with us and our listeners.

Yeah, it was really amazing to hear about your project and it was. I learned a lot definitely from hearing you speak about it. So thank you for coming on.

Thank you so much for having me, Simon Roshank. It's been great.

Yeah, so we wish you the best of luck in continuing your endeavors in natural language processing. And yeah, we hope everything works out.

Thank you. Have a great day. And yeah, thank you.

Thank you.

Have a great day.

Thanks for listening to this episode of the Wolfram Student Podcast. We hope you learned something new and took away insights from today's discussion. If you want to be featured in a future episode, fill out the form@tinyurl.comwsp-interest form so this concludes season one of the Wolfram Student Podcast. We'll be back in February with even more exciting and enriching content for all of you.

And once again, thanks for listening to this episode and the entire season of the Wolfram Student Podcast.