Python Tutorial: Basic feature extraction

In this video, we will learn to extract certain basic features from text. While not very powerful, they can give us a good idea of the text we are dealing with. The most basic feature we can extract from text is the number of characters, including white spaces. For instance, the string I don't know has 13 characters. The number of characters is the length of the string. Python gives us a built in len function which returns the length of the string passed into it. The output will be 13 here too. If our dataframe df has a textual feature, say Review, we can compute the number of characters for each review and store it as a new feature numcas by using the Pandas dataframe apply method. This is done by creating DF and assigning it to DF apply len. Another feature we can compute is the number of words. Assuming that every word is separated by a space, we can use a string's split method to convert it into a list where every element is a word. In this example, the string Mary had a little lamb is split to create a list containing the words Mary had a little and lamb. We can now compute the number of words by computing the number of elements in this list using lamb. To do this for a textual feature in a dataframe, we first define a function that takes in a string as an argument and returns the number of words in it. The steps followed inside the function are similar as before. We then pass this function wordcount into Apply. We create dfnumwords and assign it to DFU applyWordCount. Let's now compute the average length of words in a string. Let's define a function AVG wordlength, which takes in a string and returns the average wordlength. We first split the string into words and compute the length of each word. Next, we compute the average word length by dividing the sum of the lengths of all words by the number of words. We can now pass this into Apply to generate an average word length feature. Like before, when working with data such as tweets, it may be useful to compute the number of hashtags or mentions used. This tweet by Datacamp, for instance has one mention upendra35 which begins with an and two hashtags pyspark and spark, which begin with a hash. Let's write a function that computes the number of hashtags in a string. We split the string into words. We then use List Comprehension to create a list containing only those words that that are hashtags. We do this using the Startswith method of strings to find out if a word begins with hash. The final step is to return the number of elements in this list using len. The procedure to compute number of mentions is identical, except that we check if a word starts with let's see this function in action when we pass a string aigndo this is my first tweet first tweet happy. The function returns 2, which is indeed the number of hashtags in the string. There are other basic features we can compute, such as the number of sentences, number of paragraphs, number of words starting with an uppercase, all capital words, numeric quantities and etc. The procedure to do this is extremely similar to the ones we've already covered. That's enough theory for now.