Handle the difference between training and inference correctly, a classic source of bugs.
What to build
A program that runs a one-feature batch normalization layer over several training batches and then normalizes test values with the running statistics.
Requirements
- The running mean starts at 0 and the running variance at 1. For each training batch: print the normalized batch (ε = 0.00001, γ = 1, β = 0) and then update the running mean and the running variance (population variance of the batch) with the given momentum.
- Then print
running mean M, running variance Vand the normalized test values, all with four decimal places.
Input
Line 1: the momentum. Line 2: b. Then b lines, each a training batch. Last line: the test values.
Output
b + 2 lines.
Constraints
b ≤ 20; batches have 2 to 100 values.
Example
Input:
0.5
1
2 4
3
Output:
-1.0000 1.0000
running mean 1.5000, running variance 1.0000
1.5000
Why it works
At inference there is no batch to average over, so the layer uses averages of the statistics it saw during training, which makes its output depend only on the example itself.
Examples
0.5 1 2 4 3
-1.0000 1.0000 running mean 1.5000, running variance 1.0000 1.5000
0.9 2 0 2 0 2 1 2
-1.0000 1.0000 -1.0000 1.0000 running mean 0.1900, running variance 1.0000 0.8100 1.8100
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
During training, normalize with the current batch's statistics and update running = momentum × running + (1 − momentum) × batch statistic, for the mean and the variance.
Hint 2
At inference, normalize every example with the running mean and running variance only.
Solution
Show one correct solution (try it yourself first)
import math
momentum = float(input())
run_mean, run_var = 0.0, 1.0
for _ in range(int(input())):
batch = list(map(float, input().split()))
mean = sum(batch) / len(batch)
var = sum((v - mean) ** 2 for v in batch) / len(batch)
print(" ".join(f"{(v - mean) / math.sqrt(var + 0.00001) + 0.0:.4f}" for v in batch))
run_mean = momentum * run_mean + (1 - momentum) * mean
run_var = momentum * run_var + (1 - momentum) * var
test = list(map(float, input().split()))
print(f"running mean {run_mean:.4f}, running variance {run_var:.4f}")
print(" ".join(f"{(v - run_mean) / math.sqrt(run_var + 0.00001) + 0.0:.4f}" for v in test))