Cleaning and shaping data

Handling missing values, duplicates and inconsistent entries, reshaping between wide and long tables, and working with dates and text.

About 6 hours of study

Learning objectives

Lessons

  1. 1

    Cleaning data Intermediate

    Finding and fixing the problems real data has: missing values, inconsistent text, impossible values and duplicates, and the choices between dropping, flagging and filling, with mean, median or a default.

  2. 2

    Tidy data: reshaping tables Intermediate

    The tidy data principles (each variable a column, each observation a row, each value a cell), melting wide tables into long ones, pivoting long tables into wide ones, and splitting columns that hold two variables.

  3. 3

    Dates, times and text Intermediate

    Parsing dates and times with datetime and strptime, extracting parts such as the weekday, measuring differences with timedelta, and normalising free text so that the same answer written differently is counted once.

Module practice

Module quiz

1. Two rows differ only in capital letters of an email. They are…
2. Turning month columns into one month column and one value column is…
3. Which Python module parses and compares dates?
4. Why log how many rows each cleaning step removes?
5. When is filling missing values with the mean risky?
6. Why store dates as YYYY-MM-DD text when you must store them as text?

‹ Getting started with dataSummarising and combining ›