Box plots and quartiles

Intermediate

Introduction

A histogram shows one distribution in detail. When you need to compare many, such as test scores per school, delivery times per city or salaries per department, a histogram for each takes too much room. The box plot, invented by John Tukey, summarises a distribution in a handful of numbers drawn as a box and two whiskers, so that dozens of groups fit side by side.

How it works

Sort the data. Quartiles split it into quarters: Q1 has a quarter of the values below it, the median half, Q3 three quarters. With an even number of values the quartiles fall between two values, and there are several conventions for interpolating. NumPy and pandas use linear interpolation by default: the quantile p sits at position (n − 1) × p in the sorted list, and between two positions the value is interpolated proportionally. For 1 to 8, Q1 is at position 1.75, three quarters of the way from 2 to 3, so 2.75.

The interquartile range (IQR) is Q3 − Q1, the spread of the middle half. Tukey's fences sit 1.5 IQR below Q1 and 1.5 IQR above Q3. Values beyond them are outliers, drawn as separate points; the whiskers run from the box to the most extreme values still inside the fences. The box itself spans Q1 to Q3, with a line at the median.

A box plot shows the centre (median), the spread (box width and whiskers), the skew (a median off-centre in the box, or one whisker much longer than the other) and the outliers. It hides the shape inside the box: two peaks, or gaps, are invisible, which is why it complements rather than replaces a histogram.

Box plots come into their own when comparing groups. Drawn side by side on one shared axis, sorted by median, they show at a glance which groups are higher, which vary more and which have unusual values. With small groups, show the individual points too, because five numbers from ten values say little.

In Matplotlib, plt.boxplot([scores_a, scores_b]) draws two box plots with the same 1.5 IQR rule, and np.percentile(data, [25, 50, 75]) computes the quartiles with linear interpolation.

Example

# A box plot: quartiles, fences, whiskers and outliers, drawn as one line of text.
scores = [52, 61, 64, 67, 70, 71, 73, 75, 78, 80, 84, 97, 31]


def quantile(s, p):
    pos = (len(s) - 1) * p  # linear interpolation, as NumPy and pandas do by default
    i = int(pos)
    if i + 1 == len(s):
        return s[i]
    return s[i] + (pos - i) * (s[i + 1] - s[i])


s = sorted(scores)
q1, median, q3 = quantile(s, 0.25), quantile(s, 0.5), quantile(s, 0.75)
iqr = q3 - q1
low_fence, high_fence = q1 - 1.5 * iqr, q3 + 1.5 * iqr
inside = [v for v in s if low_fence <= v <= high_fence]
outliers = [v for v in s if v < low_fence or v > high_fence]
print(f"Q1 {q1:g}, median {median:g}, Q3 {q3:g}, IQR {iqr:g}")
print(f"fences {low_fence:g} and {high_fence:g}; whiskers {inside[0]} to {inside[-1]}; outliers {outliers}")

W = 41
lo, hi = s[0], s[-1]


def pos(v):
    return int((v - lo) / (hi - lo) * (W - 1) + 0.5)


line = [" "] * W
for i in range(pos(inside[0]), pos(inside[-1]) + 1):
    line[i] = "-"
for i in range(pos(q1), pos(q3) + 1):
    line[i] = "="
line[pos(q1)], line[pos(q3)] = "[", "]"
line[pos(inside[0])] = line[pos(inside[-1])] = "+"
line[pos(median)] = "|"
for v in outliers:
    line[pos(v)] = "o"
print("".join(line))
Open in compiler

Line by line

  1. quantile places p at position (n − 1) × p and interpolates between the neighbours.
  2. Q1, the median and Q3 of the 13 scores are 64, 71 and 78, so the IQR is 14.
  3. The fences are 64 − 21 = 43 and 78 + 21 = 99; only 31 falls outside, so it is an outlier.
  4. The drawing maps values to 41 columns, draws the whiskers with -, the box with = between [ and ], the median with |, whisker ends with + and the outlier with o.

Expected result

Q1 64, median 71, Q3 78, IQR 14
fences 43 and 99; whiskers 52 to 97; outliers [31]
o            +------[===|===]-----------+

In the real world

Scientists compare treatment and control groups with box plots, operations teams compare response times per data centre, and educators compare scores per class. Analysts use the 1.5 IQR rule as a first filter for suspicious values before cleaning data, always checking whether an "outlier" is an error or a real, important case.

Common mistakes

  • Mixing quartile conventions between tools and wondering why the numbers differ slightly.
  • Deleting outliers automatically, when they may be the most interesting values.
  • Using box plots for tiny groups, where five numbers misrepresent a few points.
  • Reading a box plot as showing the shape, when two peaks or gaps are hidden.

Best practices

  • Say which quartile method you use, and use the same one throughout.
  • Show individual points with the box when groups are small.
  • Sort groups by median and draw them on one shared axis.

Summary

Quartiles split sorted data into quarters; NumPy and pandas compute them by linear interpolation at position (n − 1) × p. The IQR is Q3 − Q1, Tukey's fences lie 1.5 IQR beyond the box, values beyond them are outliers, and whiskers reach the most extreme values inside. Box plots compare many groups compactly.

Quick check

1. What does the box of a box plot span?
2. Q1 is 10 and Q3 is 20. Where is the upper fence?
3. Where does a whisker end?
4. What can a box plot not show?

Key takeaways

  • A box spans Q1 to Q3 with a line at the median; whiskers reach the most extreme values inside the fences.
  • Values more than 1.5 IQR beyond the box are drawn separately as outliers.
  • Box plots are best for comparing distributions of several groups side by side.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.