Introduction
Once each column is understood on its own, the next questions are about pairs: do students who study longer score higher? Do they sleep less? Relationships are where most interesting findings come from, and also where the most common errors of reasoning are made. This lesson computes relationships and, just as importantly, shows how they mislead.
How it works
Seeing a relationship. A scatter plot puts one dot per row, one column on each axis. A cloud rising to the right is a positive relationship; falling, a negative one; a round blob, none; a curve, a relationship that is not a straight line.
Measuring it. Pearson's correlation r (from the Statistics & Probability course) is Σ(x − x̄)(y − ȳ) ÷ √(Σ(x − x̄)² × Σ(y − ȳ)²), always between −1 and 1. With several numeric columns, compute r for every pair: the result is a correlation matrix. Sorting the pairs by the size of r, ignoring the sign, shows the strongest relationships, which are candidates to study further.
Confounders. Two variables may move together because a third one drives both. Cities with more police have more crime, because bigger cities have more of both. A correlation that comes from a confounder, or from pure chance among many tested pairs, is a spurious correlation. With 20 columns there are 190 pairs, and some will look strong by luck.
Simpson's paradox. Combining groups can reverse a trend. In a well-known study of kidney stone treatments, treatment A had the higher success rate for small stones (93% against 87%) and for large stones (73% against 69%), yet treatment B looked better overall (83% against 78%). The reason: A was given mostly to the hard, large-stone cases, which lowered its overall rate. The group, stone size, was a confounder. The lesson: compare within groups as well as overall, and ask how the groups were formed.
The only reliable way to show cause is an experiment that assigns the treatment at random, so that confounders are balanced between the groups.
Example
# Relationships between columns: a correlation for every pair.
hours = [1, 2, 3, 4, 5, 6]
score = [52, 55, 61, 64, 70, 73]
sleep = [8, 7, 7, 6, 6, 5]
def corr(xs, ys):
mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
sxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
sxx = sum((x - mx) ** 2 for x in xs)
syy = sum((y - my) ** 2 for y in ys)
return round(sxy / (sxx * syy) ** 0.5, 12)
print("hours-score:", round(corr(hours, score), 3))
print("hours-sleep:", round(corr(hours, sleep), 3))
print("score-sleep:", round(corr(score, sleep), 3))Line by line
- Three columns for six students: hours studied, test score and hours of sleep.
corrcomputes Pearson's r from the means and the three sums of deviations.- Each print shows one pair of the correlation matrix.
Expected result
hours-score: 0.995
hours-sleep: -0.968
score-sleep: -0.94
Study hours rise with scores and fall with sleep. Does less sleep raise scores? Almost certainly not: studying is the likely confounder behind the negative score–sleep correlation.
In the real world
Medical research is full of correlations later overturned by randomised trials, such as hormone therapy once linked with fewer heart attacks. Admissions data at a famous university once seemed to show bias against women overall, while most departments admitted women at equal or higher rates: Simpson's paradox, because women applied more to competitive departments. Analysts in business use correlation matrices to choose which variables to put in a model.
Common mistakes
- Concluding cause from correlation, especially in observational data.
- Testing many pairs and reporting the strongest without allowing for chance.
- Comparing only the combined totals when groups of different difficulty are mixed.
- Ignoring curved relationships because r is close to 0.
Best practices
- Look at a scatter plot before trusting a correlation.
- Compare results inside meaningful groups and overall, and explain any reversal.
- Treat correlations as questions; confirm causes with experiments.
Summary
Correlation measures how two variables move together, and a correlation matrix shows every pair at once. Confounders, chance and Simpson's paradox can all create or reverse relationships, so compare within groups, plot the data, and keep claims of cause for randomised experiments.
Quick check
Key takeaways
- r between −1 and 1 measures straight-line association; compute it for every pair to find candidates.
- Correlation is not causation: look for confounders and test with experiments.
- Always compare results within groups as well as overall; combining groups can reverse a trend.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.