Cleaning and shaping data
Handling missing values, duplicates and inconsistent entries, reshaping between wide and long tables, and working with dates and text.
About 6 hours of study
Learning objectives
- Fix missing, duplicate and inconsistent values
- Reshape tables between wide and long form
- Parse dates and normalise text
Lessons
- 1
Cleaning data Intermediate
Finding and fixing the problems real data has: missing values, inconsistent text, impossible values and duplicates, and the choices between dropping, flagging and filling, with mean, median or a default.
- 2
Tidy data: reshaping tables Intermediate
The tidy data principles (each variable a column, each observation a row, each value a cell), melting wide tables into long ones, pivoting long tables into wide ones, and splitting columns that hold two variables.
- 3
Dates, times and text Intermediate
Parsing dates and times with datetime and strptime, extracting parts such as the weekday, measuring differences with timedelta, and normalising free text so that the same answer written differently is counted once.
Module practice
- Data science: Fill in missing valuesEasy
- Data science: Remove duplicate recordsMedium
- Data science: Clean a table and report the fixesHard
- Data science: Melt a wide tableEasy
- Data science: Pivot a long tableMedium
- Data science: Tidy a messy columnHard
- Data science: Which day of the week?Easy
- Data science: How long did delivery take?Medium
- Data science: Normalise free-text answersHard