Diagnose a network the way practitioners do: by measuring its activations.
What to build
A program that runs a batch through a network and reports the mean and spread of the values at every layer.
Requirements
- Layers use the format of the forward-pass exercise (
m n activation, m weight rows, biases), with activationsrelu,tanh,sigmoidorlinear. Statistics are over all values of a layer for all examples in the batch. - Print
layer K: mean M, std S(four decimal places) for the inputs (K = 0) and after each layer.
Input
Line 1: N and the number of inputs. Then N input lines. Then L and the layers.
Output
L + 1 lines.
Constraints
N ≤ 100; L ≤ 10; layers have at most 20 neurons.
Example
Input:
2 2
1 -1
2 0
2
2 2 relu
1 1
1 -1
0 0
1 2 linear
1 1
0
Output:
layer 0: mean 0.5000, std 1.1180
layer 1: mean 1.5000, std 0.8660
layer 2: mean 3.0000, std 1.0000
Why it works
If the spread of the activations falls towards zero or grows rapidly from layer to layer, the initialization or architecture needs fixing before training can work well.
Examples
2 2 1 -1 2 0 2 2 2 relu 1 1 1 -1 0 0 1 2 linear 1 1 0
layer 0: mean 0.5000, std 1.1180 layer 1: mean 1.5000, std 0.8660 layer 2: mean 3.0000, std 1.0000
2 1 1 3 1 1 1 linear 2 0
layer 0: mean 2.0000, std 1.0000 layer 1: mean 4.0000, std 2.0000
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Collect every activation value of a layer over the whole batch, then compute the mean and the population standard deviation (divide by the count).
Hint 2
Report the inputs as layer 0, then apply the layers one by one.
Solution
Show one correct solution (try it yourself first)
import math
F = {
"relu": lambda v: max(0.0, v),
"tanh": math.tanh,
"sigmoid": lambda v: 1 / (1 + math.exp(-v)),
"linear": lambda v: v,
}
N, d = map(int, input().split())
batch = [list(map(float, input().split())) for _ in range(N)]
layers = []
for _ in range(int(input())):
m, n, activation = input().split()
W = [list(map(float, input().split())) for _ in range(int(m))]
b = list(map(float, input().split()))
layers.append((W, b, F[activation]))
def report(k, rows):
values = [v for row in rows for v in row]
mean = sum(values) / len(values)
std = math.sqrt(sum((v - mean) ** 2 for v in values) / len(values))
print(f"layer {k}: mean {mean + 0.0:.4f}, std {std:.4f}")
report(0, batch)
for k, (W, b, f) in enumerate(layers, start=1):
batch = [[f(sum(w * x for w, x in zip(row, a)) + bi) for row, bi in zip(W, b)] for a in batch]
report(k, batch)