Scatter plots and correlation

Intermediate

Introduction

Do taller people weigh more? Do pricier houses sit closer to the city centre? Questions about how two numerical variables relate are answered by a scatter plot: one point per item, placed by its two values. The eye picks up direction, strength, curves, clusters and outliers immediately, and a little arithmetic then puts a number on what it sees.

How it works

Each point gets a column from its x value and a row from its y value, using the linear scale formula on each axis separately. Screen rows count from the top, so the row index is flipped: the largest y goes on the top line.

When many points share a cell, overplotting hides how many there are: a hundred customers at the same spot look like one. On a character grid you can write the count instead of a single mark; graphical libraries use transparency, smaller points, small random offsets ("jitter"), or a two-dimensional histogram of counts.

Pearson's correlation coefficient r summarises how closely the points follow a straight line:

r = sxy / (sxx * syy) ** 0.5

where sxy is the sum of products of the deviations from the means, and sxx and syy are the sums of squared deviations. r runs from −1 (a perfect falling line) through 0 (no straight-line relation) to 1 (a perfect rising line). Common rough labels are strong above 0.7 in size, moderate from 0.3, and weak below. r only measures straight lines: a perfect U shape can have r = 0. Always look at the plot.

The least-squares line has slope sxy / sxx and passes through the point of means. Each point's residual is its observed y minus the y the line predicts; large residuals mark points the line explains poorly. R squared, which for one variable equals r², is the share of the variation in y that the line explains.

Finally, correlation is not causation. Ice cream sales and drownings both rise in summer; neither causes the other. A scatter plot shows that two things move together, not why.

In Matplotlib, plt.scatter(heights, weights, alpha=0.5) draws semi-transparent points, np.corrcoef(x, y)[0, 1] computes r, and np.polyfit(x, y, 1) returns the slope and intercept of the least-squares line.

Example

# A scatter plot on a character grid, with the correlation and the least-squares line.
heights = [150, 155, 160, 162, 165, 170, 172, 175, 180, 185]  # cm
weights = [50, 54, 58, 57, 62, 66, 65, 70, 74, 80]  # kg
W, H = 30, 8

grid = [["."] * W for _ in range(H)]
for x, y in zip(heights, weights):
    col = int((x - min(heights)) / (max(heights) - min(heights)) * (W - 1) + 0.5)
    row = int((y - min(weights)) / (max(weights) - min(weights)) * (H - 1) + 0.5)
    grid[H - 1 - row][col] = "*"  # row 0 is the top line on screen
for line in grid:
    print("|" + "".join(line))
print("+" + "-" * W)

n = len(heights)
mx, my = sum(heights) / n, sum(weights) / n
sxy = sum((x - mx) * (y - my) for x, y in zip(heights, weights))
sxx = sum((x - mx) ** 2 for x in heights)
syy = sum((y - my) ** 2 for y in weights)
slope = sxy / sxx
print(f"r = {sxy / (sxx * syy) ** 0.5:.3f}")
print(f"weight = {my - slope * mx:.1f} + {slope:.3f} x height")
Open in compiler

Line by line

  1. The grid is 8 rows of 30 dots; each person becomes a * at the column of their height and the row of their weight.
  2. H - 1 - row flips the row so that heavier people appear higher up.
  3. The sums of deviations give r and the slope of the least-squares line.
  4. The intercept comes from the line passing through the means.

Expected result

|.............................*
|.........................*....
|.....................*........
|.................**...........
|............*.................
|........*.*...................
|....*.........................
|*.............................
+------------------------------
r = 0.990
weight = -76.0 + 0.834 x height

In the real world

Scientists plot dose against response, economists plot income against life expectancy, and engineers plot load against failure. Data scientists inspect scatter plots of every pair of features before modelling, and residual plots after fitting, to find patterns the model missed.

Common mistakes

  • Reading causation into correlation, without considering a third factor or chance.
  • Ignoring overplotting, which makes crowded regions look sparse.
  • Trusting r without the plot, missing curves, clusters or a single influential point.
  • Fitting a straight line to an obviously curved pattern.

Best practices

  • Plot first, then compute r and fit a line only if the pattern looks straight.
  • Use transparency or counts for large datasets.
  • Look at the residuals: they should show no pattern.

Summary

A scatter plot places each item by two numerical values, with rows flipped for the screen and overlaps counted or made transparent. Pearson's r measures straight-line association from −1 to 1; the least-squares line and its residuals describe and test the relationship. Correlation shows association, never causation by itself.

Quick check

1. What does r = −0.9 mean?
2. A perfect U-shaped pattern has r close to what?
3. What is a residual?
4. Why use transparency in a scatter plot of a million points?

Key takeaways

  • Each point is placed by its x and y on their own scales; screen rows count from the top.
  • Count or make transparent points that overlap, or the plot understates dense areas.
  • r measures straight-line association only, and correlation does not show causation.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.