Hypothesis testing and p-values

You flip a coin 20 times and get 16 heads. Is it biased, or just lucky?

The method

1. Null hypothesis H₀: nothing special is going on. Here: the coin is fair, p = 0.5.
2. Alternative H₁: the coin is not fair.
3. Compute the p-value: if H₀ were true, how likely is a result at least this extreme?
4. If the p-value is below a threshold chosen in advance, usually α = 0.05, reject H₀: the result is statistically significant.

from math import comb

n, heads = 20, 16
def prob(k):
    return comb(n, k) / 2 ** n

one_side = sum(prob(k) for k in range(heads, n + 1))
p_value = 2 * one_side
print("P(16 or more heads) =", round(one_side, 4))
print("two-sided p-value =", round(p_value, 4))
print("reject 'fair' at 0.05?", p_value < 0.05)

With a fair coin, 16 or more heads (or 4 or fewer, equally extreme the other way) happens with probability about 0.012. That is below 0.05, so we reject "the coin is fair".

What a p-value is not

• It is not the probability that H₀ is true.
• "Not significant" does not prove H₀; the sample may simply be too small.
• Testing many things at once makes some "significant" results by pure chance: with α = 0.05, about 1 in 20 tests of true null hypotheses.

Mark lesson complete