Bayesian networks

Advanced

Introduction

A full joint distribution over 30 true/false variables needs about a billion numbers, far too many to store or to estimate. Yet most variables directly affect only a few others. A Bayesian network draws those direct influences as arrows and keeps one small table per variable. It is a compact, readable model of uncertainty, used in medical diagnosis, fault finding, spam filtering and genetics.

How it works

Each node is a random variable; an arrow from A to B means A directly influences B. Each node has a conditional probability table (CPT) giving P(node | parents) for every combination of its parents' values. In the example, Cloudy influences Sprinkler and Rain, and both influence Wet.

The network defines the full joint distribution as a product:

P(x₁, …, xₙ) = ∏ P(xᵢ | parents(xᵢ))

so the probability of one complete situation is just one entry from each table, multiplied.

Inference by enumeration answers P(X | e): for each value of X, sum the product over every combination of the hidden variables (those neither queried nor observed), then normalize. It is exact but exponential in the number of hidden variables; algorithms such as variable elimination and sampling do better on large networks.

The graph also shows independence. A path between two variables is blocked by the evidence if it contains:

  • a chain A → B → C or a fork A ← B → C whose middle node B is observed; or
  • a collider A → B ← C where neither B nor any of its descendants is observed.

If every path is blocked, the variables are d-separated: independent given the evidence. The collider rule produces explaining away: Sprinkler and Rain are independent given Cloudy, but once the grass is seen wet, learning that it rained makes the sprinkler less likely, because the rain already explains the wet grass.

Syntax

def joint(world):
    return product(p(node, world[node], world) for node in order)

def query(x, evidence):
    weights = {v: sum(joint({**evidence, x: v, **h}) for h in all_hidden_assignments) for v in (True, False)}
    return weights[True] / (weights[True] + weights[False])

Example

import itertools

network = {
    "Cloudy": ([], {(): 0.6}),
    "Sprinkler": (["Cloudy"], {(True,): 0.2, (False,): 0.5}),
    "Rain": (["Cloudy"], {(True,): 0.75, (False,): 0.15}),
    "Wet": (["Sprinkler", "Rain"], {(True, True): 0.97, (True, False): 0.85, (False, True): 0.85, (False, False): 0.02}),
}
order = list(network)


def p(node, value, world):
    parents, table = network[node]
    p_true = table[tuple(world[q] for q in parents)]
    return p_true if value else 1 - p_true


def joint(world):
    result = 1.0
    for node in order:
        result *= p(node, world[node], world)
    return result


def query(x, evidence):
    totals = {True: 0.0, False: 0.0}
    hidden = [n for n in order if n != x and n not in evidence]
    for x_value in (True, False):
        for values in itertools.product((True, False), repeat=len(hidden)):
            world = {**evidence, x: x_value, **dict(zip(hidden, values))}
            totals[x_value] += joint(world)
    return totals[True] / (totals[True] + totals[False])


world = {"Cloudy": True, "Sprinkler": False, "Rain": True, "Wet": True}
print(f"P(cloudy, no sprinkler, rain, wet) = {joint(world):.4f}")
print(f"P(Rain) = {query('Rain', {}):.4f}")
print(f"P(Rain | Wet) = {query('Rain', {'Wet': True}):.4f}")
print(f"P(Sprinkler | Wet) = {query('Sprinkler', {'Wet': True}):.4f}")
print(f"P(Sprinkler | Wet, Rain) = {query('Sprinkler', {'Wet': True, 'Rain': True}):.4f}")
Open in compiler

Line by line

  1. Each node lists its parents and its CPT: the probability of being true for each combination of parent values.
  2. joint multiplies one entry per node, giving the probability of a complete situation.
  3. query sums the joint over the hidden variables for each value of the query, then normalizes.
  4. Seeing wet grass raises the probability of rain well above its prior.
  5. Seeing wet grass also makes the sprinkler more likely, but once we also know it rained, the sprinkler becomes less likely again: explaining away.

Expected result

P(cloudy, no sprinkler, rain, wet) = 0.3060
P(Rain) = 0.5100
P(Rain | Wet) = 0.7181
P(Sprinkler | Wet) = 0.4592
P(Sprinkler | Wet, Rain) = 0.2599

In the real world

Medical expert systems such as Pathfinder, for lymph-node diseases, were built as Bayesian networks. Printer and car manufacturers use them to suggest the most likely fault from symptoms, and Microsoft's Windows troubleshooters once used them. Genetic pedigree analysis is inference in a Bayesian network of inheritance.

Common mistakes

  • Drawing arrows from effects to causes, which needs more and stranger tables.
  • Forgetting that a CPT row gives P(true), so P(false) is 1 minus it.
  • Assuming two causes stay independent once their common effect is observed.
  • Leaving out normalization after enumeration.

Best practices

  • Draw arrows from causes to effects; the tables are then smaller and easier to estimate.
  • Check that each CPT row is a valid probability and that the graph has no cycles.
  • Test inference code on a two-node network you can work out by hand.

Summary

A Bayesian network is a graph with one conditional probability table per node; the product of the tables is the full joint distribution. Enumeration sums that product over hidden variables and normalizes. D-separation reads independence from the graph: observed middle nodes block chains and forks, and observed colliders open paths, causing explaining away.

Quick check

1. What does each node of a Bayesian network store?
2. How is the probability of a complete assignment computed?
3. In A → B ← C with nothing observed, are A and C independent?
4. Why does knowing it rained lower P(sprinkler | wet grass)?

Key takeaways

  • P(x₁, …, xₙ) = the product of P(xᵢ | parents(xᵢ)): a few small tables replace one huge one.
  • Any query can be answered by summing that product over the unknown variables.
  • Evidence on a common cause or a middle link blocks a path; evidence on a common effect opens it.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.