A survey data cleaner
The problem
Clean a messy survey export with free-text answers, inconsistent categories, invalid dates and duplicates, and produce a tidy long table ready for analysis.
Requirements
- Text normalisation and recoding of category answers with map
- Dates parsed with errors='coerce' and invalid ones counted
- Duplicate responses removed after normalisation
- A melt from one column per question to a long table, with a summary by question
Technologies
- Python 3
- pandas
- The .str and .dt accessors
- melt and groupby
Step by step
- Profile the raw export: types, missing values and odd answers.
- Write the cleaning steps as functions and chain them with pipe.
- Reshape to long format and summarise each question.
- Write a cleaning log with the row count after each step.
Expected result
A tidy dataset and a cleaning log that explains every change.
How it is evaluated
- Each cleaning rule is a small tested function
- No row is dropped without being counted
- The long table has one answer per row
- The summary matches hand counts on a sample
Ways to improve it
- Detect free-text answers that mean the same thing
- Add response time from timestamps
- Save the clean data as Parquet