Deep learning: Activation statistics per layer

Diagnose a network the way practitioners do: by measuring its activations.

What to build

A program that runs a batch through a network and reports the mean and spread of the values at every layer.

Requirements

  • Layers use the format of the forward-pass exercise (m n activation, m weight rows, biases), with activations relu, tanh, sigmoid or linear. Statistics are over all values of a layer for all examples in the batch.
  • Print layer K: mean M, std S (four decimal places) for the inputs (K = 0) and after each layer.

Input

Line 1: N and the number of inputs. Then N input lines. Then L and the layers.

Output

L + 1 lines.

Constraints

N ≤ 100; L ≤ 10; layers have at most 20 neurons.

Example

Input:

2 2
1 -1
2 0
2
2 2 relu
1 1
1 -1
0 0
1 2 linear
1 1
0

Output:

layer 0: mean 0.5000, std 1.1180
layer 1: mean 1.5000, std 0.8660
layer 2: mean 3.0000, std 1.0000

Why it works

If the spread of the activations falls towards zero or grows rapidly from layer to layer, the initialization or architecture needs fixing before training can work well.

Examples

Example 1 · Input
2 2
1 -1
2 0
2
2 2 relu
1 1
1 -1
0 0
1 2 linear
1 1
0
Expected output
layer 0: mean 0.5000, std 1.1180
layer 1: mean 1.5000, std 0.8660
layer 2: mean 3.0000, std 1.0000
Example 2 · Input
2 1
1
3
1
1 1 linear
2
0
Expected output
layer 0: mean 2.0000, std 1.0000
layer 1: mean 4.0000, std 2.0000

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Collect every activation value of a layer over the whole batch, then compute the mean and the population standard deviation (divide by the count).

Hint 2

Report the inputs as layer 0, then apply the layers one by one.

Solution

Show one correct solution (try it yourself first)
import math

F = {
    "relu": lambda v: max(0.0, v),
    "tanh": math.tanh,
    "sigmoid": lambda v: 1 / (1 + math.exp(-v)),
    "linear": lambda v: v,
}
N, d = map(int, input().split())
batch = [list(map(float, input().split())) for _ in range(N)]
layers = []
for _ in range(int(input())):
    m, n, activation = input().split()
    W = [list(map(float, input().split())) for _ in range(int(m))]
    b = list(map(float, input().split()))
    layers.append((W, b, F[activation]))


def report(k, rows):
    values = [v for row in rows for v in row]
    mean = sum(values) / len(values)
    std = math.sqrt(sum((v - mean) ** 2 for v in values) / len(values))
    print(f"layer {k}: mean {mean + 0.0:.4f}, std {std:.4f}")


report(0, batch)
for k, (W, b, f) in enumerate(layers, start=1):
    batch = [[f(sum(w * x for w, x in zip(row, a)) + bi) for row, bi in zip(W, b)] for a in batch]
    report(k, batch)

Lesson: Initialization and vanishing gradients →