Introduction
Text columns are where data is messiest: names typed in every case, extra spaces, product codes that hide several facts in one string, free-text comments. pandas gives every Series of strings a .str accessor with Python's string methods and regular expressions, applied to the whole column at once, with missing values handled for you.
How it works
Cleaning. Each method returns a new Series, so they chain:
s = pd.Series([" ana SMITH ", "Ben Lee"])
s.str.strip().str.title().tolist() # ['Ana Smith', 'Ben Lee']
s.str.len().tolist() # [12, 7]
Missing values stay missing: a string method returns NaN for them, and a numeric result such as str.len() becomes a float column because of the NaN. str.replace("old", "new") replaces literal text in pandas 2; pass regex=True for patterns. To collapse repeated spaces, use s.str.split().str.join(" ") or s.str.replace(r"\s+", " ", regex=True). Cleaning before comparing matters: "ANA SMITH" and "ana smith " are the same person to a reader, but not to == or drop_duplicates.
Testing. str.contains("late", case=False, na=False) gives a boolean mask for filtering. Two parameters matter: case=False ignores capitals, and na=False makes missing values count as no match; without it, the mask contains NaN and filtering with it raises an error. Note that str.contains treats its pattern as a regular expression by default, so pass regex=False to search for text such as "1+1". str.startswith and str.endswith test the ends.
Splitting. s.str.split(" ") gives lists; s.str.split(" ", expand=True) gives a DataFrame with one column per piece, and s.str.split().str[0] takes the first piece of each.
Extracting. Many columns pack several facts into one code, such as "ABC-123-XL" (category, number, size). str.extract takes a regular expression with capture groups and returns one column per group:
codes = pd.Series(["ABC-123-XL", "bad"])
codes.str.extract(r"^([A-Z]+)-(\d+)-(S|M|L|XL)$")[0].tolist() # ['ABC', nan]
Rows that do not match get NaN in every column, which makes invalid codes easy to count. In the pattern, ^ and $ anchor the start and end, [A-Z]+ means one or more capital letters, \d+ one or more digits, and (S|M|L|XL) one of the listed sizes.
Example
# The .str accessor modelled on a list: clean, test and split text values.
names = [" ana SMITH ", "Ben Lee", None, "cy o'neil"]
clean = [" ".join(n.split()).title() if n is not None else None for n in names]
print("cleaned:", clean)
print("contains 'lee':", [("lee" in n.lower()) if n is not None else False for n in clean])
print("first names:", [n.split()[0] if n is not None else None for n in clean])
print("lengths:", [len(n) if n is not None else None for n in clean])Line by line
" ".join(n.split())collapses runs of spaces and trims the ends;title()fixes the case.NonestaysNone, as NaN does in pandas.- The containment test lowers the text first and treats a missing value as no match, like
case=False, na=False. split()[0]takes the first word, like.str.split().str[0].
Expected result
cleaned: ['Ana Smith', 'Ben Lee', None, "Cy O'Neil"]
contains 'lee': [False, True, False, False]
first names: ['Ana', 'Ben', None, 'Cy']
lengths: [9, 7, None, 9]
In the real world
Customer lists are deduplicated after normalising names and emails. Support tickets are filtered by keywords, product codes and addresses are split into fields, and log lines are parsed with str.extract into tables that can be grouped and counted.
Common mistakes
- Comparing text before normalising case and spaces.
- Filtering with a mask that contains NaN, because na=False was not given.
- Forgetting that str.contains uses regular expressions, so characters like +, ( or . behave specially.
- Writing patterns without ^ and $, which match part of a longer invalid string.
Best practices
- Normalise text columns once, early, and keep the raw column for reference.
- Always pass case and na to str.contains in filters.
- Anchor extraction patterns and count the rows that did not match.
Summary
The .str accessor applies string methods to a whole column and leaves missing values missing. Clean with strip, title, lower and replace; filter with str.contains using case=False and na=False; split with expand=True; and pull fields out of codes with str.extract, whose capture groups become columns with NaN for non-matching rows.
Quick check
Key takeaways
- String methods on a column go through .str and skip missing values.
- Pass case=False and na=False to str.contains when filtering.
- str.extract turns each capture group into a column, NaN where the pattern does not match.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.