Text basics
What NLP is and the text pipeline, tokenization and normalization, and regular expressions for finding and cleaning text.
About 4 hours of study
Learning objectives
- Describe NLP tasks and the stages of a text pipeline
- Tokenize and normalize text, knowing what each step loses
- Extract and replace patterns with regular expressions
Lessons
- 1
What natural language processing is Intermediate
What NLP is, why human language is hard for computers, and the pipeline almost every NLP system follows: raw text, sentences, tokens, counts, features and a model. Build a tiny pipeline in plain Python.
- 2
Tokenization and normalization Intermediate
Split raw text into tokens and clean them up: separate punctuation, expand contractions, lowercase, remove stopwords and cut words to their stems. See what each step gains and what it can destroy, such as the word "not".
- 3
Regular expressions for text Intermediate
Find, extract and replace patterns in text with Python's re module: character classes, quantifiers, groups, anchors and alternation. Pull out emails, dates and hashtags, and write a regular-expression tokenizer.
Module practice
- NLP: Characters, words and sentencesEasy
- NLP: Most frequent wordsMedium
- NLP: Split text into sentencesHard
- NLP: Split words from punctuationEasy
- NLP: Normalize and drop stopwordsMedium
- NLP: A suffix-stripping stemmerHard
- NLP: Hashtags and mentionsEasy
- NLP: Find and normalize datesMedium
- NLP: Redact personal dataHard