Probability and Bayes' rule

Intermediate

Introduction

Logic deals in certainties, but agents rarely have them. Sensors are noisy, tests are imperfect and the world is only partly visible. Probability gives a precise way to hold degrees of belief and to change them when evidence arrives. Spam filters, medical decision aids, self-driving cars and speech recognisers all rest on the rules in this lesson, above all Bayes' rule.

How it works

A random variable has a set of possible values, each with a probability; the probabilities add up to 1. For several variables, the joint distribution gives the probability of every combination. From it you can compute anything:

  • marginal: P(rain) = sum of P(rain, road) over every road state;
  • conditional: P(rain | traffic) = P(rain, traffic) / P(traffic).

The product rule rearranges the conditional: P(A, B) = P(A | B) P(B) = P(B | A) P(A). Equating the two gives Bayes' rule:

P(H | E) = P(E | H) P(H) / P(E)

where P(E) = P(E | H) P(H) + P(E | not H) P(not H). Diagnosis is the typical use: a test's makers know P(positive | disease), but a doctor needs P(disease | positive).

Normalization is a shortcut. Compute P(E | h) P(h) for every hypothesis h; these weights are proportional to the posteriors, so divide each by their total.

Two variables are independent if P(A, B) = P(A) P(B). Two pieces of evidence are conditionally independent given H if, once H is known, one tells you nothing about the other. Then evidence can be applied one piece at a time: the posterior after the first test is the prior for the second.

The base-rate effect surprises people: with 1% prevalence, a 90% sensitive test with 5% false positives is right only about 15% of the time when it says "positive", because healthy people vastly outnumber sick ones.

Syntax

p_e = p_e_given_h * p_h + p_e_given_not_h * (1 - p_h)
posterior = p_e_given_h * p_h / p_e
weights = {h: likelihood[h] * prior[h] for h in prior}          # normalization
posterior = {h: w / sum(weights.values()) for h, w in weights.items()}

Example

prior = 0.01            # 1 person in 100 has the condition
sensitivity = 0.90      # P(positive | condition)
false_positive = 0.05   # P(positive | no condition)

p_positive = sensitivity * prior + false_positive * (1 - prior)
posterior = sensitivity * prior / p_positive
print(f"P(positive) = {p_positive:.4f}")
print(f"P(condition | positive) = {posterior:.4f}")

people = 10_000
sick = people * prior
healthy = people - sick
print(f"out of {people} people: {sick * sensitivity:.0f} sick and {healthy * false_positive:.0f} healthy people test positive")

p_again = sensitivity * posterior + false_positive * (1 - posterior)
print(f"after a second positive test: {sensitivity * posterior / p_again:.4f}")

joint = {
    ("rain", "traffic"): 0.20,
    ("rain", "no traffic"): 0.10,
    ("dry", "traffic"): 0.20,
    ("dry", "no traffic"): 0.50,
}
p_rain = sum(p for (weather, _), p in joint.items() if weather == "rain")
p_traffic = sum(p for (_, road), p in joint.items() if road == "traffic")
print(f"P(rain) = {p_rain:.4f}")
print(f"P(rain | traffic) = {joint[('rain', 'traffic')] / p_traffic:.4f}")
Open in compiler

Line by line

  1. The test finds 90% of real cases but also flags 5% of healthy people, and 1% of people have the condition.
  2. P(positive) is the total of both ways to test positive: about 5.85%.
  3. Bayes' rule gives only a 15.38% chance of the condition after one positive test.
  4. Counting people makes it clear: 90 sick people test positive, but so do 495 healthy ones.
  5. A second independent positive test, starting from the new belief, raises the chance to 76.60%. The joint table at the end gives P(rain) = 0.3 and P(rain | traffic) = 0.5.

Expected result

P(positive) = 0.0585
P(condition | positive) = 0.1538
out of 10000 people: 90 sick and 495 healthy people test positive
after a second positive test: 0.7660
P(rain) = 0.3000
P(rain | traffic) = 0.5000

In the real world

Spam filters weigh words with Bayes' rule. Robots localize themselves by repeatedly multiplying their belief by the probability of what their sensors report and normalizing. Screening programmes for rare diseases follow up every positive with a second, different test, for exactly the reason shown in the example.

Common mistakes

  • Confusing P(E | H) with P(H | E): "the test detects 90% of cases" is not "a positive is 90% likely to be a case".
  • Ignoring the prior, the base-rate fallacy.
  • Forgetting to normalize, so probabilities do not add up to 1.
  • Applying two pieces of evidence as if independent when they are not, such as the same test repeated on the same sample.

Best practices

  • Write every probability with its condition: P(positive | sick), never just "accuracy".
  • Check results with counts out of 10,000; they are easier to reason about.
  • Use normalization when there are many hypotheses; it avoids computing P(E) separately.

Summary

The joint distribution determines every marginal and conditional probability. Bayes' rule turns the probability of evidence given a cause into the probability of the cause given the evidence, and needs the prior. Beliefs are updated by multiplying by the likelihood of each new piece of evidence and normalizing.

Quick check

1. How is P(A | B) defined?
2. Why is P(condition | positive) only about 15% in the example?
3. What does normalizing do?
4. What is P(rain) if P(rain, traffic) = 0.2 and P(rain, no traffic) = 0.1?

Key takeaways

  • Any probability about a set of variables can be computed from their joint distribution.
  • Bayes' rule needs the prior: a test for a rare condition gives mostly false positives.
  • Normalizing (dividing by the total) turns unnormalized weights into probabilities.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.