Introduction
Even with good initialization, the scale of the values inside a network drifts as the weights change during training: each layer has to keep adapting to inputs whose scale keeps moving. Normalization layers fix the scale at every step. Batch normalization (2015) let image networks train many times faster; layer normalization is part of every transformer.
How it works
Standardizing a list of numbers subtracts their mean μ and divides by their standard deviation:
x̂ = (x − μ) / √(σ² + ε)
The result has mean 0 and standard deviation (almost exactly) 1. The small ε, such as 0.00001, prevents division by zero when all values are equal.
Normalizing everything to mean 0 and spread 1 could remove useful information, so normalization layers add two learned parameters per feature: y = γ x̂ + β. If the network needs the original scale, it can learn γ = σ and β = μ.
The two main kinds differ in what they average over. For a batch of N examples with d features:
- Batch normalization normalizes each feature (column) over the N examples in the batch.
- Layer normalization normalizes each example (row) over its d features.
Batch norm depends on the other examples in the batch, which is a problem for small batches and for sequences of different lengths. At inference there may be only one example, so during training batch norm keeps running averages of the means and variances (running = momentum × running + (1 − momentum) × batch value) and uses those instead. Layer norm needs no batch at all and behaves the same in training and inference, which is why transformers and recurrent networks use it.
Syntax
mean = sum(column) / n
var = sum((v - mean) ** 2 for v in column) / n
normalized = [(v - mean) / math.sqrt(var + 1e-5) for v in column]
output = [gamma * v + beta for v in normalized]
Example
import math
batch = [
[2.0, 100.0, -1.0],
[4.0, 300.0, -3.0],
[6.0, 200.0, -2.0],
[8.0, 400.0, -6.0],
]
EPS = 1e-5
def batch_norm(rows, gamma, beta):
n, d = len(rows), len(rows[0])
out = [[0.0] * d for _ in range(n)]
for j in range(d):
column = [row[j] for row in rows]
mean = sum(column) / n
var = sum((v - mean) ** 2 for v in column) / n
for i in range(n):
out[i][j] = gamma[j] * (rows[i][j] - mean) / math.sqrt(var + EPS) + beta[j]
return out
def layer_norm(row):
mean = sum(row) / len(row)
var = sum((v - mean) ** 2 for v in row) / len(row)
return [(v - mean) / math.sqrt(var + EPS) for v in row]
print("batch norm (each feature over the batch):")
for row in batch_norm(batch, [1.0, 1.0, 1.0], [0.0, 0.0, 0.0]):
print(" ", " ".join(f"{v:7.3f}" for v in row))
print("batch norm with gamma = 2 and beta = 5 for the first feature:")
print(" ", [round(row[0], 3) for row in batch_norm(batch, [2.0, 1.0, 1.0], [5.0, 0.0, 0.0])])
print("layer norm (each example over its features):")
for row in batch:
print(" ", " ".join(f"{v:7.3f}" for v in layer_norm(row)))Line by line
- The batch has 4 examples and 3 features on very different scales: units, hundreds and negatives.
- Batch norm normalizes each column separately, so every feature ends up with mean 0 and spread 1.
- With γ = 2 and β = 5, the first feature's normalized values are doubled and shifted to centre on 5.
- Layer norm normalizes each row instead: each example's three values are compared only with each other.
- Because the second feature is so large, each row's layer-norm result is dominated by it, which shows why the choice of axis matters.
Expected result
batch norm (each feature over the batch):
-1.342 -1.342 1.069
-0.447 0.447 0.000
0.447 -0.447 0.535
1.342 1.342 -1.604
batch norm with gamma = 2 and beta = 5 for the first feature:
[2.317, 4.106, 5.894, 7.683]
layer norm (each example over its features):
-0.675 1.414 -0.739
-0.682 1.414 -0.732
-0.664 1.413 -0.750
-0.670 1.414 -0.744
In the real world
ResNet and most image networks put batch norm after each convolution. Transformers such as GPT and BERT use layer norm (or its simpler cousin RMSNorm) around every attention and feed-forward block. Forgetting to switch batch norm to inference mode, so that it keeps using batch statistics, is a classic deployment bug.
Common mistakes
- Normalizing over the wrong axis: columns for batch norm, rows for layer norm.
- Leaving out ε, which divides by zero when all values are equal.
- Using batch statistics at inference instead of the running averages.
- Using the sample variance (divide by n − 1) where the standard uses n.
Best practices
- Check after normalization that each group has mean about 0 and spread about 1.
- Prefer layer norm for sequences and small batches.
- Keep γ and β learnable; they let the network choose the best scale.
Summary
Normalization subtracts a mean and divides by a standard deviation, then applies a learned scale γ and shift β. Batch norm normalizes each feature over the batch and needs running statistics at inference; layer norm normalizes each example over its features and works the same everywhere.
Quick check
Key takeaways
- Normalization keeps every layer's inputs on a similar scale, so training is faster and less sensitive to initialization.
- Batch norm averages over the batch for each feature; layer norm averages over the features for each example.
- A small ε inside the square root avoids dividing by zero.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.