Processing text data

Introduction

Real data usually arrives as text: a file, a form, a message. The core skill is splitting that text into pieces, converting the pieces, and putting results back together.

Most data begins life as plain text: exports from spreadsheets, logs from servers, messages from people. The steps are nearly always the same: split it up, convert the pieces, then work with real values.

How it works

Words

text = "the quick brown fox jumps over the lazy dog the end"
words = text.split()
print(len(words), "words")
print(words.count("the"), "times the word the")

split() with no argument splits on any run of spaces and drops empty pieces. words.count("the") counts whole words. (Note that text.count("the") would count the letters inside words such as "other" too.)

Back to text with join

parts = ["2026", "10", "29"]
date = "-".join(parts)
print(date)
print(date.replace("-", "/"))

"-".join(parts) glues a list of strings together with - between them. It is the reverse of split.

Syntax

lines = text.splitlines()
name, mark = line.split(",")
words = text.split()
", ".join(words)

Example

data = "Rahim,72\nNadia,91\nKarim,85"
rows = []
for line in data.splitlines():
    name, mark = line.split(",")
    rows.append((name, int(mark)))
print(rows)
print(f"Average: {sum(m for _, m in rows) / len(rows):.2f}")

splitlines() breaks the text into lines, and split(",") breaks each line into fields. Every field is still text, so int(mark) converts the mark before storing it. This comma-separated layout is called CSV, and it is how spreadsheets export data.

Line by line

1. data has three lines, separated by \n.
2. splitlines() gives one item for each line.
3. line.split(",") breaks a line in two, and the two names receive the parts.
4. int(mark) converts the mark, and the pair is added to rows.
5. The last line adds up the marks and shows the average with two decimals.

Expected result

[('Rahim', 72), ('Nadia', 91), ('Karim', 85)]
Average: 82.67

In the real world

Reading a spreadsheet export, cleaning survey answers and scanning a server's log are daily tasks for anyone working with data. For complicated files, Python's csv module handles details such as commas inside fields.

Common mistakes

• Forgetting to convert: "72" + "85" is "7285".
• Unpacking a line that has the wrong number of fields raises a ValueError; check len(fields) when data might be messy.
• Calling join on numbers: convert them first, for example " ".join(str(n) for n in nums).

Best practices

• Convert fields as soon as you split them.
• Check the number of fields when the data may be messy.
• Use the csv module for files from outside your own program.

Summary

Text is processed by splitting it into lines and fields, converting them and reassembling the results. Everything from split is a string, and join puts strings back together.

Practice

Parse "apple:3;pen:10;book:2" into a dictionary of item to quantity, then print the item with the largest quantity.

Mark lesson complete