Introduction
Before a program can count, compare or learn from text, it must decide what the units of that text are. "Don't panic!" could be two tokens, three or four. Is "U.S." one token or three? Should "Cats" and "cats" count as the same word? These decisions are tokenization and normalization, and they affect every result that follows. A sentiment model that throws away "not" will read "not good" as good; a search engine that keeps "Running" and "run" apart will miss matches.
How it works
Splitting on spaces is the simplest tokenizer, but it leaves punctuation glued to words: "dogs." and "dogs" become different tokens. A better rule walks through the text one character at a time: letters, digits and apostrophes build up a word; any other character ends the word, and if it is not a space it becomes a token of its own.
Normalization then maps different forms to one:
- Lowercasing merges "The", "THE" and "the". It loses a little information ("Apple" the company versus "apple" the fruit).
- Contraction expansion turns "aren't" into "are not", so that "not" is visible to later steps.
- Stopword removal drops very common words such as "the", "a" and "of". They carry little meaning for topics, so removing them shrinks the vocabulary, but never remove words such as "not" or "no" when meaning matters.
- Stemming cuts endings with rules: "running" becomes "runn", "cats" becomes "cat". It is fast but crude, and stems need not be real words.
- Lemmatization maps a word to its dictionary form using a word list and the part of speech: "ran" becomes "run", "better" becomes "good". It is more accurate but needs knowledge of the language.
For text in many languages, there is also Unicode normalization: "café" can be stored as one character "é" or as "e" plus an accent mark, and both must compare equal (Python's unicodedata.normalize does this).
Example
text = "Cats aren't dogs. The CATS ran, running fast!"
CONTRACTIONS = {"aren't": "are not", "can't": "can not", "don't": "do not"}
STOPWORDS = ["the", "a", "an", "and", "are", "is", "of", "to"]
def tokenize(text):
tokens, word = [], ""
for ch in text + " ":
if ch.isalnum() or ch == "'":
word += ch
continue
if word:
tokens.append(word)
word = ""
if not ch.isspace():
tokens.append(ch)
return tokens
def stem(word):
for suffix in ("ing", "ed", "s"):
if word.endswith(suffix) and len(word) - len(suffix) >= 3:
return word[: -len(suffix)]
return word
tokens = tokenize(text)
words = []
for t in tokens:
if t[0].isalnum():
words += CONTRACTIONS.get(t.lower(), t.lower()).split()
kept = [w for w in words if w not in STOPWORDS]
stems = [stem(w) for w in kept]
print("split(): ", text.split())
print("tokens: ", tokens)
print("lowercase:", words)
print("no stop: ", kept)
print("stems: ", stems)
print("vocabulary:", len(set(t for t in tokens if t[0].isalnum())), "->", len(set(stems)))Line by line
text.split()keeps punctuation attached: "dogs." and "ran," are not clean words.tokenizereads character by character. Letters, digits and apostrophes extend the current word; anything else ends it, and punctuation becomes its own token.- A space is appended to the text so the last word is always closed.
- For each word token, the contraction table turns "aren't" into "are not"; everything is lowercased.
- The stopword list removes "are" and "the", but "not" stays, because it is not in the list. That is deliberate.
stemtries the endings "ing", "ed" and "s" and keeps at least three letters. "cats" becomes "cat", but "running" becomes "runn": a typical stemming error.- The vocabulary shrinks from 8 distinct word forms to 6 stems, so later counts are less spread out.
Expected result
split(): ['Cats', "aren't", 'dogs.', 'The', 'CATS', 'ran,', 'running', 'fast!']
tokens: ['Cats', "aren't", 'dogs', '.', 'The', 'CATS', 'ran', ',', 'running', 'fast', '!']
lowercase: ['cats', 'are', 'not', 'dogs', 'the', 'cats', 'ran', 'running', 'fast']
no stop: ['cats', 'not', 'dogs', 'cats', 'ran', 'running', 'fast']
stems: ['cat', 'not', 'dog', 'cat', 'ran', 'runn', 'fast']
vocabulary: 8 -> 6
In the real world
Production tokenizers handle hundreds of special cases: abbreviations, URLs, emojis, hashtags and numbers such as "3.5". spaCy, for example, does it like this (shown, not run: StepVioz's Python runs in your browser without these libraries; install Python and pip install spacy to try it):
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Don't panic! The U.S. team's runners ran 3.5 km.")
print([token.text for token in doc])
print([token.lemma_ for token in doc])
It splits "Don't" into "Do" and "n't", keeps "U.S." and "3.5" whole, and lemmatizes "ran" to "run". Modern language models skip most normalization and use sub-word tokenizers instead, which you will meet later in this course.
Common mistakes
- Removing negations as stopwords, which turns "not good" into "good".
- Lowercasing before named-entity recognition, which hides the capital letters that signal names.
- Applying a stemmer and then showing stems to users: "runn" is not a word.
- Using an English stopword list on other languages.
Best practices
- Decide each normalization step for your task: topic models like stopword removal; sentiment and translation usually do not.
- Keep the original text next to the tokens, so you can always show or recover it.
- Test the tokenizer on awkward cases: abbreviations, numbers, contractions, quotes and empty input.
- Use the same tokenizer for training and for prediction; mismatches silently reduce accuracy.
Summary
Tokenization splits text into words, numbers and punctuation; splitting on spaces alone leaves punctuation attached. Normalization (lowercasing, expanding contractions, removing stopwords, stemming or lemmatizing) merges forms that mean the same thing and shrinks the vocabulary, but every step can also destroy information, so choose them to fit the task.
Quick check
Key takeaways
- Good tokens make every later step better; bad tokens quietly ruin counts and models.
- Each normalization step merges forms that should be the same, but can also merge forms that should stay different.
- Stemmers are fast and crude; lemmatizers are slower and need knowledge of the language.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.