R Tutorial: Text cleaning basics

If you have ever heard the phrase garbage in garbage out when creating a model. The same applies with text analysis. We just learned how to tokenize which can really expose potential garbage in our text. Let's take the next step after tokenization and create better input text so we get better analysis. Before we look at some simple pre processing steps to clean our data, I'd like to introduce a second data set we will be exploring. 538 recently published A Ton of public data. One of these data sets consisted of almost 3 million Russian troll tweets. These are tweets from bots that tweeted during the 2016 US election cycle. We will explore the first 20,000 tweets as well as use some of the metadata such as the number of followers, number following publish date and account type to aid in some of our analysis. This is a great data set for topic modeling, classification tasks named entity recognition and others. You can imagine tweets probably have a lot of garbage to show this. Look at the most common words in the troll tweet dataset. First we tokenize by words and then we count how often these words occur. The results are not that surprising. T co is a shorthand for Twitter's web address and was probably picked up when these tweets were scraped from the web. HTTPs has a similar story, but none of our top four occurring words are helpful. We need to remove them. Removing stop words with the tidytext package takes just one additional command. Tidytext Anti join function will remove a tibble of words from a column of text. The typical entry in this tibble is the word you want to remove and the lexicon or source for where the word came from. Anti join will return the original tibble with all stop words removed from the text column. Note that stopwords is a tibble of common words provided by the tidytext package. Let's look at the results a second time. Ok, so T Co, HTTPs and HTTP are still problematic, but we finally have two interesting top words, Black Lives Matter and Trump. We will not get political in this course, but these are still interesting results. We still need to work on those first common words we can add to our tibble of stop words or create our own. Here I am adding three stop words to the stop words HTTPs, HTTP and T Co. We can run through the process of removing stop words and counting the word occurrences one last time. We get some interesting results. Within the first 20,000 tweets, these seven words occurred the most often. One additional step I want to cover is called stimming. Stimming is the process of transforming words into their root. For example, both enlisted and enlisting would be trimmed to their root enlist. This is an important step when trying to really understand which words are being used. We will use the word stem function from the snowballc package as it works extremely well with the tidy principles. Consider this example. We want to take our tidy tweets and perform a mutation. This mutation will stem the words using the wordstim function. Notice here that matter was trimmed to mat even though it was part of a much larger word. And cop was the seventh most common word before stimming, but now it has jumped to second. Let's.