Deep learning: Watch a gradient vanish or explode

Trace how much of a gradient survives the trip back through many layers.

What to build

A program that computes the gradient of the last activation of a chain of one-neuron layers with respect to every earlier activation.

Requirements

  • Activations: sigmoid (slope a(1 − a)), tanh (slope 1 − a²), relu or linear (slope 1). Each layer has one weight and no bias.
  • Print layer K: G for K from L down to 0, with G in scientific notation with three decimals (Python's .3e).

Input

Line 1: the activation and x. Line 2: the weights w₁ … w_L.

Output

L + 1 lines.

Constraints

L ≤ 100.

Example

Input:

tanh 0
0.5 0.5 0.5

Output:

layer 3: 1.000e+00
layer 2: 5.000e-01
layer 1: 2.500e-01
layer 0: 1.250e-01

Why it works

Backpropagation multiplies the gradient by each layer's weight and slope, so many small factors shrink it towards zero and many large ones blow it up.

Examples

Example 1 · Input
tanh 0
0.5 0.5 0.5
Expected output
layer 3: 1.000e+00
layer 2: 5.000e-01
layer 1: 2.500e-01
layer 0: 1.250e-01
Example 2 · Input
linear 1
2 2 2 2
Expected output
layer 4: 1.000e+00
layer 3: 2.000e+00
layer 2: 4.000e+00
layer 1: 8.000e+00
layer 0: 1.600e+01

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Forward: a₀ = x, zₖ = wₖ aₖ₋₁, aₖ = f(zₖ); keep every z.

Hint 2

Backward: g_L = 1, then gₖ₋₁ = gₖ × f′(zₖ) × wₖ. ReLU's slope is 1 for z > 0 and 0 otherwise.

Solution

Show one correct solution (try it yourself first)
import math

activation, x = input().split()
a = float(x)
ws = list(map(float, input().split()))
f = {
    "sigmoid": lambda z: 1 / (1 + math.exp(-z)),
    "tanh": math.tanh,
    "relu": lambda z: max(0.0, z),
    "linear": lambda z: z,
}[activation]
slopes = []
for w in ws:
    z = w * a
    a = f(z)
    if activation == "sigmoid":
        slopes.append(a * (1 - a))
    elif activation == "tanh":
        slopes.append(1 - a * a)
    elif activation == "relu":
        slopes.append(1.0 if z > 0 else 0.0)
    else:
        slopes.append(1.0)
g = 1.0
print(f"layer {len(ws)}: {g:.3e}")
for k in range(len(ws), 0, -1):
    g = g * slopes[k - 1] * ws[k - 1]
    print(f"layer {k - 1}: {g + 0.0:.3e}")

Lesson: Initialization and vanishing gradients →