Descriptive statistics

Intermediate

Introduction

A column of a thousand numbers cannot be read; a few well-chosen numbers can describe it. Descriptive statistics summarise a column by where its values lie (the centre) and how far they spread. They are the first thing to compute for every numeric variable, and they also expose problems: an average quiz time of 16 minutes when most people take 13 suggests something unusual in the data.

How it works

Measures of centre. The mean is the sum divided by the count. The median is the middle value of the sorted data, or the average of the two middle values when the count is even. The mode is the most common value, mostly useful for categories and small whole numbers. When the data has extreme values, the mean is pulled towards them and the median is not; average income is far above median income in most countries for exactly that reason.

Measures of spread. The range is the maximum minus the minimum, which depends on just two values. The variance is the average squared distance from the mean, and the standard deviation is its square root, in the same units as the data. For a sample, divide by n − 1 instead of n (statistics.stdev); for a whole population, divide by n (statistics.pstdev).

Quartiles cut the sorted data into quarters. A simple method: Q2 is the median; Q1 is the median of the lower half and Q3 the median of the upper half, leaving out the middle value when the count is odd. Other methods exist and give slightly different answers, so state yours. The interquartile range, IQR = Q3 − Q1, measures the spread of the middle half and ignores extremes. Together, the minimum, Q1, the median, Q3 and the maximum form the five-number summary, which a box plot draws.

A common rule calls a value an outlier if it is below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR. An outlier is not automatically an error: it may be a typo, or the most interesting observation in the data. Investigate it, and report results with and without it.

Python's statistics module has mean, median, mode, stdev, pstdev and quantiles. In pandas, df["col"].describe() prints the count, mean, standard deviation, minimum, quartiles and maximum at once.

Example

# Descriptive statistics: centre, spread, quartiles and outliers of one column.
import statistics

times = [12, 15, 11, 14, 12, 45, 12, 16, 14, 13]  # minutes to finish a quiz
print("count:", len(times))
print("mean:", statistics.mean(times))
print("median:", statistics.median(times))
print("mode:", statistics.mode(times))
print("range:", max(times) - min(times))
print("standard deviation:", round(statistics.stdev(times), 2))

s = sorted(times)
half = len(s) // 2
q1 = statistics.median(s[:half])
q3 = statistics.median(s[half:] if len(s) % 2 == 0 else s[half + 1:])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print("quartiles:", q1, q3, "IQR:", iqr)
print("outliers:", [t for t in times if t < low or t > high])
print("mean without outliers:", round(statistics.mean([t for t in times if low <= t <= high]), 1))
Open in compiler

Line by line

  1. Ten quiz times, one of them, 45, far longer than the rest.
  2. The mean is pulled up by 45; the median and the mode stay near the typical time.
  3. The quartiles come from the medians of the lower and upper halves of the sorted list.
  4. The fences at 1.5 × IQR beyond the quartiles mark 45 as an outlier.
  5. Without it, the mean drops close to the median.

Expected result

count: 10
mean: 16.4
median: 13.5
mode: 12
range: 34
standard deviation: 10.17
quartiles: 12 15 IQR: 3
outliers: [45]
mean without outliers: 13.2

In the real world

Government statistics report median household income rather than the mean, because a few very high incomes would distort the picture. Quality control in factories tracks the standard deviation of product sizes, and investigates outliers as possible faults. Box plots of the five-number summary are a standard way to compare groups in reports and papers.

Common mistakes

  • Reporting only the mean for data with extreme values.
  • Deleting outliers without investigating them, which can remove the most important data.
  • Mixing sample and population standard deviation without saying which was used.
  • Comparing spreads in different units, such as grams and kilograms, without converting.

Best practices

  • Report a centre and a spread together, such as the median and the IQR.
  • State the method used for quartiles and outliers.
  • Look at the five-number summary of every numeric column before modelling it.

Summary

The mean, median and mode describe a column's centre, and the range, variance, standard deviation and IQR its spread. Quartiles split sorted data into quarters, the 1.5 × IQR rule flags outliers, and the median and IQR are the robust choice when extreme values are present.

Quick check

1. Which measure of centre is least affected by one very large value?
2. What does the IQR measure?
3. With Q1 = 12 and Q3 = 15, which value is an outlier by the 1.5 × IQR rule?
4. Why divide by n − 1 for a sample's variance?

Key takeaways

  • Describe a column by its centre (mean or median) and its spread (standard deviation or IQR).
  • One extreme value moves the mean and the standard deviation a lot, and the median and IQR hardly at all.
  • Flag outliers with a stated rule, investigate them, and report results with and without them.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.