The data science workflow

Advanced

Introduction

The previous lessons covered the steps one by one: reading, cleaning, reshaping, summarising, relating, predicting and evaluating. Real projects chain them together, run them many times as data changes, and hand the results to people who make decisions. This lesson is about doing that well: making the work repeatable, honest and safe.

How it works

Pipelines. Write the analysis as a pipeline of small steps, each a function: load → clean → transform → analyse → report. Never edit the raw data by hand; keep it unchanged and let code produce everything else. Then, when next month's data arrives or a mistake is found, the whole analysis reruns in seconds and gives a reproducible result. Fix random seeds for the same reason.

Logging cleaning decisions. Each cleaning rule changes the data: trimming spaces, fixing case, dropping rows with a missing amount, removing duplicates. Count what each step did ("7 rows read, 2 dropped for a missing amount, 1 duplicate removed, 4 kept") and report it. A reader who sees that half the rows were dropped will read the results very differently.

Splits and leakage. Split training and test data in a fixed, documented way, such as every fifth row or a fixed seed. Then make sure nothing from the test data reaches the model. A subtle case is standardising: if the mean and standard deviation used to scale the data are computed on all rows, the test rows have influenced the training. Compute them on the training rows only and apply the same numbers to the test rows. The same applies to time: a model that predicts tomorrow must not be trained with data from after tomorrow. This mistake is called data leakage, and it makes models look far better than they will be in use.

Communicating. A result should state the question, the data used and its limits, the method, the answer with its uncertainty, and what would change the conclusion. One clear chart beats ten tables, and "sales rose 4% ± 3%" is more honest than "sales rose".

Ethics and privacy. Data is often about people. Collect only what the question needs, remove names and identifiers when they are not needed, follow the law on personal data, and ask whether a model could treat groups unfairly, for example because past decisions in the training data were biased.

Example

# A whole analysis in miniature: load, clean, summarise, report.
raw = """region,amount
North,120
south,80
North,
East,45
SOUTH,60
north,30"""

lines = raw.split("\n")[1:]
totals, dropped = {}, 0
for line in lines:
    region, amount = line.split(",")
    if amount == "":
        dropped += 1
        continue
    region = region.title()
    totals[region] = totals.get(region, 0) + float(amount)
print("rows:", len(lines), " dropped:", dropped)
for region in sorted(totals, key=totals.get, reverse=True):
    print(f"{region}: {totals[region]:.2f}")
Open in compiler

Line by line

  1. raw stands in for a file: a header and six rows with messy region names and a missing amount.
  2. Each row is split; a row with a missing amount is counted and skipped.
  3. title() turns north, NORTH and North into one spelling.
  4. The report states how many rows were read and dropped, then the totals from largest to smallest.

Expected result

rows: 6  dropped: 1
North: 150.00
South: 140.00
East: 45.00

In the real world

Companies run pipelines every night that rebuild dashboards from raw data. Scientific journals increasingly ask for the code and data behind results so others can reproduce them. Laws such as the EU's GDPR set rules for personal data, and several public scandals came from models trained on biased historical decisions.

Common mistakes

  • Editing data by hand in a spreadsheet, so the analysis cannot be repeated.
  • Scaling or cleaning with statistics from all rows, leaking test information.
  • Hiding dropped rows, which can bias the result without anyone knowing.
  • Reporting a single number without its uncertainty or limits.

Best practices

  • Keep raw data read-only and generate everything else with code.
  • Log the row count after every cleaning step.
  • Fit every transformation on training data only.
  • Write results for the decision maker: question, answer, uncertainty, limits.

Summary

A good analysis is a pipeline: code that turns raw data into a report and can be rerun at any time. It logs every cleaning decision, keeps test data and future data out of training, protects people's data, and communicates the answer with its uncertainty and limits.

Quick check

1. Why keep the raw data unchanged and clean it with code?
2. Standardising with the mean of all rows before splitting causes…
3. Why report how many rows were dropped in cleaning?
4. Which is the most honest way to report a result?

Key takeaways

  • Write analyses as pipelines of small steps that can be rerun from raw data.
  • Compute every statistic a model uses from training data only.
  • Report what was dropped and why, with uncertainty and limits, and protect people's data.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.