Deep learning: One training step of a network

Do a full forward pass, backward pass and update for a two-layer network.

What to build

A program that performs one step of gradient descent for a network with one ReLU hidden layer and one linear output.

Requirements

  • The network maps n inputs to h hidden ReLU units to one output. ReLU's derivative is 1 for a positive input and 0 otherwise (including 0). The loss is (ŷ − y)² for one example; every parameter p becomes p − rate × ∂L/∂p.
  • Print loss L (before the update), then the updated W1: (rows separated by |), b1:, W2: and b2:, with four decimal places.

Input

Line 1: n and h. Line 2: x. Then h rows of W₁, then b₁, then W₂, then a line with b₂, y and the learning rate.

Output

Five lines.

Constraints

n, h ≤ 10.

Example

Input:

2 2
1 2
1 0
0 1
0 -3
1 1
0 3 0.1

Output:

loss 4.0000
W1: 1.4000 0.8000 | 0.0000 1.0000
b1: 0.4000 -3.0000
W2: 1.4000 1.0000
b2: 0.4000

Why it works

Backpropagation reuses the forward values to find every weight's effect on the loss in one backward pass, and gradient descent then moves each weight a small step to reduce the loss.

Examples

Example 1 · Input
2 2
1 2
1 0
0 1
0 -3
1 1
0 3 0.1
Expected output
loss 4.0000
W1: 1.4000 0.8000 | 0.0000 1.0000
b1: 0.4000 -3.0000
W2: 1.4000 1.0000
b2: 0.4000
Example 2 · Input
1 1
2
0.5
0
2
1 2 0.25
Expected output
loss 1.0000
W1: -1.5000
b1: -1.0000
W2: 1.5000
b2: 0.5000

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Forward: z₁ = W₁x + b₁, h = ReLU(z₁), ŷ = W₂·h + b₂, loss = (ŷ − y)². Keep z₁ and h.

Hint 2

Backward: g = 2(ŷ − y); dW₂ = g·h, db₂ = g; dz₁ᵢ = g·W₂ᵢ if z₁ᵢ > 0 else 0; dW₁ᵢⱼ = dz₁ᵢ·xⱼ, db₁ = dz₁. Compute every gradient before updating any weight.

Solution

Show one correct solution (try it yourself first)
n, h = map(int, input().split())
x = list(map(float, input().split()))
W1 = [list(map(float, input().split())) for _ in range(h)]
b1 = list(map(float, input().split()))
W2 = list(map(float, input().split()))
b2, y, rate = map(float, input().split())

z1 = [sum(w * xi for w, xi in zip(row, x)) + bi for row, bi in zip(W1, b1)]
hidden = [max(0.0, z) for z in z1]
y_hat = sum(w * a for w, a in zip(W2, hidden)) + b2
g = 2 * (y_hat - y)

dW2 = [g * a for a in hidden]
db2 = g
dz1 = [g * w if z > 0 else 0.0 for w, z in zip(W2, z1)]
dW1 = [[d * xi for xi in x] for d in dz1]
db1 = dz1

W1 = [[w - rate * dw for w, dw in zip(row, drow)] for row, drow in zip(W1, dW1)]
b1 = [b - rate * d for b, d in zip(b1, db1)]
W2 = [w - rate * d for w, d in zip(W2, dW2)]
b2 = b2 - rate * db2


def fmt(values):
    return " ".join(f"{v + 0.0:.4f}" for v in values)


print(f"loss {(y_hat - y) ** 2:.4f}")
print("W1: " + " | ".join(fmt(row) for row in W1))
print("b1: " + fmt(b1))
print("W2: " + fmt(W2))
print(f"b2: {b2:.4f}")

Lesson: Backpropagation →