Text basics

What NLP is and the text pipeline, tokenization and normalization, and regular expressions for finding and cleaning text.

About 4 hours of study

Learning objectives

Lessons

  1. 1

    What natural language processing is Intermediate

    What NLP is, why human language is hard for computers, and the pipeline almost every NLP system follows: raw text, sentences, tokens, counts, features and a model. Build a tiny pipeline in plain Python.

  2. 2

    Tokenization and normalization Intermediate

    Split raw text into tokens and clean them up: separate punctuation, expand contractions, lowercase, remove stopwords and cut words to their stems. See what each step gains and what it can destroy, such as the word "not".

  3. 3

    Regular expressions for text Intermediate

    Find, extract and replace patterns in text with Python's re module: character classes, quantifiers, groups, anchors and alternation. Pull out emails, dates and hashtags, and write a regular-expression tokenizer.

Module practice

Module quiz

1. What is a corpus?
2. Which tokenizer output keeps punctuation separate from the words?
3. What can lowercasing lose?
4. What does the regex [A-Z][a-z]+ match in 'see Paris now'?
5. What does re.sub(r'\s+', ' ', text) do?
6. Why can a stemmer map 'university' and 'universe' to the same stem?

Counting words ›