Regular expressions for text

Intermediate

Introduction

A lot of NLP work is finding text that has a predictable shape: email addresses, dates, prices, hashtags, phone numbers, or the tokens of a sentence. Writing loops that check characters one by one quickly becomes long and fragile. Regular expressions (regexes) describe such shapes in one short pattern, and Python's built-in re module finds, extracts and replaces them. Regexes are in almost every NLP pipeline, from cleaning raw text to the rules that pre-tokenize text for large language models.

How it works

A regex is made of a few building blocks:

  • Literal characters match themselves: cat matches "cat".
  • Character classes: [aeiou] matches one vowel, [A-Za-z] one letter, [^0-9] anything except a digit. Shorthands: \d (digit), \w (letter, digit or underscore), \s (whitespace), and . (any character).
  • Quantifiers repeat the item before them: * (zero or more), + (one or more), ? (optional), {4} (exactly four), {1,2} (one or two). They are greedy: they take as much as they can; add ? (as in +?) to take as little as possible.
  • Anchors match positions, not characters: ^ (start), $ (end), \b (a word boundary).
  • Groups in parentheses ( ) capture part of a match; (?: ) groups without capturing.
  • Alternation a|b matches either side.

Always write patterns as raw strings, r"\d+", so Python does not treat the backslashes itself.

The main functions:

  • re.findall(p, text): a list of every match (or of the group, if the pattern has one).
  • re.finditer(p, text): match objects with .group(), .groups(), .start() and .end().
  • re.search(p, text): the first match anywhere, or None.
  • re.sub(p, replacement, text): replace every match.

A regex tokenizer lists the token shapes as alternatives, tried left to right: a word with an optional apostrophe part, a number with an optional decimal part, or any single punctuation character.

Example

import re

text = "Email ana@uni.edu or bo.li@mail.com by 2024-03-15. Call 555-0142! #nlp #python"

print("emails:  ", re.findall(r"[\w.]+@\w+\.\w+", text))
print("dates:   ", re.findall(r"\d{4}-\d{2}-\d{2}", text))
print("phones:  ", re.findall(r"\b\d{3}-\d{4}\b", text))
print("hashtags:", re.findall(r"#(\w+)", text))

for m in re.finditer(r"(\d{4})-(\d{2})-(\d{2})", text):
    year, month, day = m.groups()
    print(f"date at position {m.start()}: day {day}, month {month}, year {year}")

print(re.sub(r"\d", "#", "Call 555-0142"))
pattern = r"[A-Za-z]+(?:'[a-z]+)?|\d+(?:\.\d+)?|[^\w\s]"
print(re.findall(pattern, "It's 3.5 km, isn't it?"))
Open in compiler

Line by line

  1. [\w.]+@\w+\.\w+ reads: name characters, @, a domain word, a literal dot, an ending. The dot in the class allows "bo.li".
  2. \d{4}-\d{2}-\d{2} matches ISO dates. \b\d{3}-\d{4}\b matches the phone number but not the end of the date, because \b needs a word boundary.
  3. #(\w+) has a group, so findall returns only the word after #.
  4. finditer gives match objects; m.groups() returns the three captured parts and m.start() the position of the match.
  5. re.sub(r"\d", "#", ...) replaces every digit, a simple way to hide numbers.
  6. The tokenizer pattern keeps "It's", "isn't" and "3.5" whole and splits off the comma and the question mark.

Expected result

emails:   ['ana@uni.edu', 'bo.li@mail.com']
dates:    ['2024-03-15']
phones:   ['555-0142']
hashtags: ['nlp', 'python']
date at position 39: day 15, month 03, year 2024
Call ###-####
["It's", '3.5', 'km', ',', "isn't", 'it', '?']

In the real world

Regexes clean scraped web text, find personal data to redact before sharing documents, extract prices and dates from emails, and split text before tokenizers such as GPT-2's byte-pair encoding run (it uses a regex to separate words, numbers and punctuation first). Search tools such as grep are regexes on the command line. For meaning, though, regexes fail: no pattern can tell whether "Apple" is the company or the fruit. That is where the statistical models of the later lessons come in.

Common mistakes

  • Forgetting the raw string: "\b" is a backspace character in Python, but r"\b" is a word boundary.
  • Unescaped dots: . matches any character, so 3.5 as a pattern also matches "335". Write 3\.5.
  • Greedy quantifiers going too far: ".*" on a line with two quoted strings matches from the first quote to the last.
  • Expecting findall to return whole matches when the pattern has groups: it returns the groups. Use (?: ) for grouping only.

Best practices

  • Build patterns step by step and test each on examples that should match and examples that should not.
  • Compile a pattern you use many times with re.compile, and give it a clear name.
  • Prefer specific classes (\d{4}) to wide ones (.*).
  • Comment complex patterns, or build them from smaller named pieces.

Summary

Regular expressions describe text patterns with literals, character classes, quantifiers, anchors, groups and alternation. Python's re module finds (findall, finditer, search) and replaces (sub) matches. They are the right tool for well-shaped strings such as dates, emails and tokens, and the wrong tool for meaning.

Quick check

1. What does the pattern \d{4} match?
2. What does re.findall return for a pattern with one group?
3. Why write patterns as raw strings such as r"\b"?
4. Which task is a poor fit for regular expressions?

Key takeaways

  • Regular expressions describe text patterns compactly and are built into Python as the re module.
  • findall returns the matches, finditer gives positions and groups, and sub replaces matches.
  • Regexes are ideal for well-shaped data such as dates and emails, but not for meaning.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.