Deep learning: A batch normalization layer

Normalize each feature over a batch, then rescale it.

What to build

A program that applies batch normalization to a batch of examples.

Requirements

  • For each feature j, compute the mean and population variance over the N examples, standardize with ε = 0.00001, then multiply by γⱼ and add βⱼ.
  • Print the N output rows, values with four decimal places separated by spaces.

Input

Line 1: N and d. Then N rows of d values. Then the d values of γ, then the d values of β.

Output

N lines.

Constraints

N, d ≤ 100.

Example

Input:

2 2
1 10
3 30
1 1
0 0

Output:

-1.0000 -1.0000
1.0000 1.0000

Why it works

Each feature is brought to mean 0 and spread 1 over the batch, so the next layer sees inputs on a steady scale, and γ and β let training choose a better scale if needed.

Examples

Example 1 · Input
2 2
1 10
3 30
1 1
0 0
Expected output
-1.0000 -1.0000
1.0000 1.0000
Example 2 · Input
3 1
1
2
3
2
5
Expected output
2.5505
5.0000
7.4495

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Work column by column: each feature has its own mean, variance, γ and β.

Hint 2

Output = γ × (x − mean) / √(variance + 0.00001) + β.

Solution

Show one correct solution (try it yourself first)
import math

N, d = map(int, input().split())
rows = [list(map(float, input().split())) for _ in range(N)]
gamma = list(map(float, input().split()))
beta = list(map(float, input().split()))
out = [[0.0] * d for _ in range(N)]
for j in range(d):
    column = [row[j] for row in rows]
    mean = sum(column) / N
    var = sum((v - mean) ** 2 for v in column) / N
    for i in range(N):
        out[i][j] = gamma[j] * (rows[i][j] - mean) / math.sqrt(var + 0.00001) + beta[j]
for row in out:
    print(" ".join(f"{v + 0.0:.4f}" for v in row))

Lesson: Batch normalization and layer normalization →