Introduction
A deep network passes its signal through many layers, and its gradient back through the same layers. Each layer multiplies the values by its weights. If those multiplications shrink the values a little, after 50 layers nothing is left; if they grow them a little, the values overflow. For years this made deep networks nearly impossible to train. Careful initialization, together with ReLU activations and normalization, solved most of the problem.
How it works
Consider one layer: each output is a sum of n inputs times weights. If the inputs and weights are independent with mean 0, the variance of the output is about n × Var(w) × Var(x). To keep the variance the same from layer to layer, Var(w) must be about 1/n. Too small a scale and the activations shrink towards zero layer after layer; too large and they grow without limit.
- Xavier (Glorot) initialization balances the forward signal and the backward gradient: std = √(2 / (fan-in + fan-out)), or uniform on ±√(6 / (fan-in + fan-out)). It suits tanh and sigmoid.
- He initialization accounts for ReLU zeroing about half the values: std = √(2 / fan-in).
The gradient has the same problem in reverse. Backpropagating through a layer multiplies the gradient by the weight and by the activation's slope. For a chain of layers, the gradient reaching the first layer is a product of many such factors:
∂a_L / ∂a_0 = Π w_k f′(z_k)
Sigmoid's slope is never more than 0.25, so ten sigmoid layers can shrink a gradient by a factor of a million: a vanishing gradient. Weights larger than 1 can make it explode instead. ReLU's slope is exactly 1 where it is active, which is one reason it trains deep networks so well, although a unit whose input is always negative passes no gradient at all (a "dead" ReLU).
Syntax
xavier_limit = math.sqrt(6 / (fan_in + fan_out)) # uniform(-limit, limit)
xavier_std = math.sqrt(2 / (fan_in + fan_out))
he_std = math.sqrt(2 / fan_in)
W = [[random.gauss(0, he_std) for _ in range(fan_in)] for _ in range(fan_out)]
Example
import math
import random
random.seed(5)
WIDTH, DEPTH = 32, 8
def run(scale_name, std):
x = [random.gauss(0, 1) for _ in range(WIDTH)]
spreads = []
for _ in range(DEPTH):
W = [[random.gauss(0, std) for _ in range(WIDTH)] for _ in range(WIDTH)]
x = [max(0.0, sum(w * v for w, v in zip(row, x))) for row in W]
mean = sum(x) / WIDTH
spreads.append(math.sqrt(sum((v - mean) ** 2 for v in x) / WIDTH))
print(f"{scale_name:13}", " ".join(f"{s:9.3g}" for s in spreads))
print("standard deviation of the activations after each of 8 ReLU layers:")
run("std 0.05", 0.05)
run("std 1.0", 1.0)
run("He sqrt(2/n)", math.sqrt(2 / WIDTH))Line by line
- A random input passes through 8 ReLU layers of 32 neurons, three times with different weight scales.
- After each layer, the program measures the standard deviation of the 32 activations.
- With std 0.05, the spread shrinks at every layer: the signal vanishes.
- With std 1.0, it grows several times at every layer: the signal explodes.
- With He initialization, √(2/32) = 0.25, the spread stays between about 0.5 and 1 throughout, neither vanishing nor exploding.
Expected result
standard deviation of the activations after each of 8 ReLU layers:
std 0.05 0.132 0.0245 0.00412 0.000797 9.85e-05 1.34e-05 2.38e-06 4.23e-07
std 1.0 3.66 16.5 71.5 298 1.29e+03 4.4e+03 1.78e+04 8.2e+04
He sqrt(2/n) 0.936 0.944 0.982 1.05 0.985 0.669 0.628 0.555
In the real world
Every deep learning framework initializes layers with Xavier or He schemes by default. Very deep networks add residual connections (adding a layer's input to its output) and normalization layers, which keep signals and gradients healthy even with hundreds of layers. Gradient clipping, which caps the size of the gradient, protects recurrent networks and transformers from occasional explosions.
Common mistakes
- Initializing every weight to zero, which makes all neurons identical and stops learning.
- Using one fixed scale such as 0.01 for every layer, whatever its size.
- Stacking many sigmoid layers and wondering why the early layers do not learn.
- Ignoring a loss that becomes NaN, which usually means something exploded.
Best practices
- Use He initialization for ReLU layers and Xavier for tanh or sigmoid layers.
- Print the spread of activations and gradients per layer when a deep network will not train.
- Prefer ReLU-family activations in deep stacks.
Summary
Each layer scales the signal by about the weight variance times the fan-in, so the wrong initialization scale makes signals vanish or explode with depth. Xavier and He initialization pick the scale from the layer sizes. Gradients flow back as products of weights and activation slopes, so small slopes such as sigmoid's make them vanish.
Quick check
Key takeaways
- Each layer multiplies the signal by roughly the weight scale times the fan-in; the wrong scale compounds over depth.
- He initialization, std = √(2 / fan-in), suits ReLU layers; Xavier suits tanh and sigmoid.
- Sigmoid's slope is at most 0.25, so long chains of sigmoids make gradients vanish.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.