Introduction
Two kinds of column cause trouble in almost every dataset: dates and free text. Dates arrive as text in many styles, and questions about them (which weekday is busiest, how long deliveries take) need real date values. Free-text answers such as "Mango", " mango!" and "MANGO" are the same answer written three ways, and must be made identical before they can be counted. Python's standard library handles both.
How it works
The datetime module turns text into date values with strptime and a format string. %Y is a four-digit year, %m the month, %d the day, %H the hour and %M the minute:
from datetime import datetime
t = datetime.strptime("2024-03-01 09:15", "%Y-%m-%d %H:%M")
The value has parts you can read directly (t.year, t.month, t.hour) and methods such as t.weekday(), which counts from Monday as 0. strftime formats it back to text: %A gives the weekday's name, %B the month's. ISO dates such as 2024-03-01 sort correctly even as text, which is one reason they are the preferred format for data.
Subtracting two datetimes gives a timedelta. It stores days and seconds; delta.total_seconds() / 3600 gives the duration in hours as one number. Adding a timedelta to a date gives a new date: t + timedelta(days=7) is a week later, with months and leap years handled correctly. Doing this arithmetic by hand with text is the source of many bugs.
Free text needs normalisation: a fixed series of steps that turns every variant into one standard form. Common steps are strip() to remove spaces around the text, lower() for case, removing punctuation, collapsing repeated spaces, and simple rules for plurals. The rules should be written down and applied in the same order every time; then counting is just a dictionary of counts. No fixed rules are perfect, so look at the most frequent results to check that they make sense.
In pandas, pd.to_datetime, the .dt accessor (.dt.day_name(), .dt.hour) and the .str accessor (.str.strip().str.lower()) do the same jobs on whole columns.
Example
# Dates and text: parse timestamps, measure gaps, and normalise free-text answers.
from datetime import datetime, timedelta
orders = ["2024-03-01 09:15", "2024-03-02 18:40", "2024-03-09 11:05"]
times = [datetime.strptime(t, "%Y-%m-%d %H:%M") for t in orders]
for t in times:
print(t.strftime("%A %d %B"), "at hour", t.hour)
gap = times[-1] - times[0]
print("first to last order:", gap.days, "days and", gap.seconds // 3600, "hours")
print("a week after the first order:", (times[0] + timedelta(days=7)).date())
answers = [" Mango", "mango!", "MANGO ", "Lychee", "lychee."]
clean = [a.strip().lower().strip("!.") for a in answers]
counts = {}
for a in clean:
counts[a] = counts.get(a, 0) + 1
print(counts)Line by line
- Each order time is parsed with the same explicit format.
strftimeprints the weekday and month names, andt.hourreads the hour.- Subtracting the first time from the last gives a timedelta with days and seconds.
- Adding seven days gives a date a week later.
- The answers are stripped, lower-cased and cleared of punctuation, so five raw answers become two values.
Expected result
Friday 01 March at hour 9
Saturday 02 March at hour 18
Saturday 09 March at hour 11
first to last order: 8 days and 1 hours
a week after the first order: 2024-03-08
{'mango': 3, 'lychee': 2}
In the real world
Delivery companies measure the time from order to delivery, call centres count calls per hour of the day, and shops compare weekend and weekday sales, all from timestamps. Survey teams and support departments normalise free-text answers and complaint categories before reporting them. Time zones and daylight-saving changes add further traps, which is why many systems store all times in UTC.
Common mistakes
- Comparing dates as text in a non-ISO format, where "02/03" and "10/01" sort wrongly.
- Computing durations from timedelta.seconds alone, which ignores whole days.
- Guessing a date's format, when 01/02 can mean 1 February or 2 January.
- Counting free text before normalising it, so one answer appears as several.
Best practices
- Store dates in ISO format (YYYY-MM-DD) and parse them with an explicit format.
- Use total_seconds() for durations, and timedelta for date arithmetic.
- Write normalisation rules once, apply them in a fixed order, and review the result.
Summary
Parse dates with datetime.strptime and an explicit format, read their parts or format them with strftime, and use subtraction and timedelta for durations and date arithmetic. Normalise free text with a fixed sequence of rules before counting it.
Quick check
Key takeaways
- Convert date text to datetime values once, with an explicit format, before any calculation.
- Subtracting datetimes gives a timedelta; total_seconds() gives the duration as one number.
- Normalise free text with strip, lower and a fixed set of rules, and count only after that.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.