Initialization and vanishing gradients

Advanced

Introduction

A deep network passes its signal through many layers, and its gradient back through the same layers. Each layer multiplies the values by its weights. If those multiplications shrink the values a little, after 50 layers nothing is left; if they grow them a little, the values overflow. For years this made deep networks nearly impossible to train. Careful initialization, together with ReLU activations and normalization, solved most of the problem.

How it works

Consider one layer: each output is a sum of n inputs times weights. If the inputs and weights are independent with mean 0, the variance of the output is about n × Var(w) × Var(x). To keep the variance the same from layer to layer, Var(w) must be about 1/n. Too small a scale and the activations shrink towards zero layer after layer; too large and they grow without limit.

  • Xavier (Glorot) initialization balances the forward signal and the backward gradient: std = √(2 / (fan-in + fan-out)), or uniform on ±√(6 / (fan-in + fan-out)). It suits tanh and sigmoid.
  • He initialization accounts for ReLU zeroing about half the values: std = √(2 / fan-in).

The gradient has the same problem in reverse. Backpropagating through a layer multiplies the gradient by the weight and by the activation's slope. For a chain of layers, the gradient reaching the first layer is a product of many such factors:

∂a_L / ∂a_0 = Π w_k f′(z_k)

Sigmoid's slope is never more than 0.25, so ten sigmoid layers can shrink a gradient by a factor of a million: a vanishing gradient. Weights larger than 1 can make it explode instead. ReLU's slope is exactly 1 where it is active, which is one reason it trains deep networks so well, although a unit whose input is always negative passes no gradient at all (a "dead" ReLU).

Syntax

xavier_limit = math.sqrt(6 / (fan_in + fan_out))     # uniform(-limit, limit)
xavier_std = math.sqrt(2 / (fan_in + fan_out))
he_std = math.sqrt(2 / fan_in)
W = [[random.gauss(0, he_std) for _ in range(fan_in)] for _ in range(fan_out)]

Example

import math
import random

random.seed(5)
WIDTH, DEPTH = 32, 8


def run(scale_name, std):
    x = [random.gauss(0, 1) for _ in range(WIDTH)]
    spreads = []
    for _ in range(DEPTH):
        W = [[random.gauss(0, std) for _ in range(WIDTH)] for _ in range(WIDTH)]
        x = [max(0.0, sum(w * v for w, v in zip(row, x))) for row in W]
        mean = sum(x) / WIDTH
        spreads.append(math.sqrt(sum((v - mean) ** 2 for v in x) / WIDTH))
    print(f"{scale_name:13}", " ".join(f"{s:9.3g}" for s in spreads))


print("standard deviation of the activations after each of 8 ReLU layers:")
run("std 0.05", 0.05)
run("std 1.0", 1.0)
run("He sqrt(2/n)", math.sqrt(2 / WIDTH))
Open in compiler

Line by line

  1. A random input passes through 8 ReLU layers of 32 neurons, three times with different weight scales.
  2. After each layer, the program measures the standard deviation of the 32 activations.
  3. With std 0.05, the spread shrinks at every layer: the signal vanishes.
  4. With std 1.0, it grows several times at every layer: the signal explodes.
  5. With He initialization, √(2/32) = 0.25, the spread stays between about 0.5 and 1 throughout, neither vanishing nor exploding.

Expected result

standard deviation of the activations after each of 8 ReLU layers:
std 0.05          0.132    0.0245   0.00412  0.000797  9.85e-05  1.34e-05  2.38e-06  4.23e-07
std 1.0            3.66      16.5      71.5       298  1.29e+03   4.4e+03  1.78e+04   8.2e+04
He sqrt(2/n)      0.936     0.944     0.982      1.05     0.985     0.669     0.628     0.555

In the real world

Every deep learning framework initializes layers with Xavier or He schemes by default. Very deep networks add residual connections (adding a layer's input to its output) and normalization layers, which keep signals and gradients healthy even with hundreds of layers. Gradient clipping, which caps the size of the gradient, protects recurrent networks and transformers from occasional explosions.

Common mistakes

  • Initializing every weight to zero, which makes all neurons identical and stops learning.
  • Using one fixed scale such as 0.01 for every layer, whatever its size.
  • Stacking many sigmoid layers and wondering why the early layers do not learn.
  • Ignoring a loss that becomes NaN, which usually means something exploded.

Best practices

  • Use He initialization for ReLU layers and Xavier for tanh or sigmoid layers.
  • Print the spread of activations and gradients per layer when a deep network will not train.
  • Prefer ReLU-family activations in deep stacks.

Summary

Each layer scales the signal by about the weight variance times the fan-in, so the wrong initialization scale makes signals vanish or explode with depth. Xavier and He initialization pick the scale from the layer sizes. Gradients flow back as products of weights and activation slopes, so small slopes such as sigmoid's make them vanish.

Quick check

1. What is the He initialization standard deviation for a layer with 50 inputs?
2. Why do long chains of sigmoid layers make gradients vanish?
3. What happens if every weight starts at zero?
4. What does ReLU's slope equal for a positive input?

Key takeaways

  • Each layer multiplies the signal by roughly the weight scale times the fan-in; the wrong scale compounds over depth.
  • He initialization, std = √(2 / fan-in), suits ReLU layers; Xavier suits tanh and sigmoid.
  • Sigmoid's slope is at most 0.25, so long chains of sigmoids make gradients vanish.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.