Python Tutorial: Introduction to tokenization

In this video, we'll learn more about string tokenization. Tokenization is the process of transforming a string or document into smaller chunks, which we call tokens. This is usually one step in the process of preparing a text for natural language processing. There are many different theories and rules regarding tokenization, and you can create your own tokenization rules using regular expressions. But normally tokenization will do things like break out words or sentences, often separate punctuation. Or you can even just tokenize parts of a string, like separating all hashtags in a tweet. One library that's commonly used for symbol tokenization is nltk, the Natural Language Toolkit Library. Here is a short example of using the Word tokenize method to break down a string into tokens. We can see from the result that words are separated and punctuation are individual tokens as well. So why bother with tokenization? Because it can help us with some simple text processing tasks like mapping parts of speech, matching common words, and perhaps removing unwanted tokens like common words or repeated words. Here we have a good example. The sentence is I don't like Sam's shoes. When we tokenize it, we can clearly see the negation in the not, and we can see possession with the apostrophe s. These indicators can help us determine meaning from simple text. Beyond just tokenizing words. NLTK has plenty of other tokenizers you can use, including these ones that you'll be working with in this chapter. The sendtokenize function will split a document into individual sentences. The regexp tokenize uses regular expressions to tokenize the string, giving you more granular control over the process. And the tweet tokenizer does neat things like recognize hashtags, mentions, and when you have too many punctuation symbols following a sentence. How convenient. You'll be using more regex in this section as well, not only when you're tokenizing, but also figuring out how to parse tokens in text using the regex modules. Rematch and Research are pretty essential for Python string processing. Learning when to use Search versus Match can be challenging, so let's take a look at how they're different when we use search and match with the same pattern and string. When the pattern is at the beginning of the string, we can see we find identical matches. This is the case with matching and searching ABCDE with the pattern abc. When we use Search for a pattern that appears later in the string, we get a result, but we don't get the same result using Match. This is because Match will try and match a string from the beginning until it cannot match any longer. Search will go through the entire string to look for match options. If you need to find a pattern that might not be at the beginning of the string, you should use Search. If you want to be specific about the composition of the entire string, or at least the initial pattern, then you should use Match. Now it's your turn to try some tokenization.