Deep learning: Gradients averaged over a batch

Compute the exact gradients that a full-batch update uses, for every parameter.

What to build

A program that computes the mean loss and all parameter gradients of a one-hidden-layer ReLU network over a batch.

Requirements

  • The network and loss are those of the previous exercise.
  • Print loss L, then dW1: (rows separated by |), db1:, dW2: and db2:, all with four decimal places.

Input

Line 1: n, h and N. Then N lines: the inputs and the target. Then h rows of W₁, then b₁, then W₂, then b₂.

Output

Five lines.

Constraints

n, h ≤ 10; N ≤ 50.

Example

Input:

1 1 2
1 2
2 4
1
0
1
0

Output:

loss 2.5000
dW1: -5.0000
db1: -3.0000
dW2: -5.0000
db2: -3.0000

Why it works

The mean loss is an average, so its gradient is the average of the single-example gradients, each found by backpropagation.

Examples

Example 1 · Input
1 1 2
1 2
2 4
1
0
1
0
Expected output
loss 2.5000
dW1: -5.0000
db1: -3.0000
dW2: -5.0000
db2: -3.0000
Example 2 · Input
2 2 1
1 2 3
1 0
0 1
0 -3
1 1
0
Expected output
loss 4.0000
dW1: -4.0000 -8.0000 | 0.0000 0.0000
db1: -4.0000 0.0000
dW2: -4.0000 0.0000
db2: -4.0000

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Run the forward pass for every example, keeping z₁ and the hidden activations, and add each example's contribution to the gradient sums.

Hint 2

With the mean squared error, each example's output gradient is g = 2(ŷ − y) / N; everything else follows the single-example rules.

Solution

Show one correct solution (try it yourself first)
n, h, N = map(int, input().split())
data = []
for _ in range(N):
    *x, y = map(float, input().split())
    data.append((x, y))
W1 = [list(map(float, input().split())) for _ in range(h)]
b1 = list(map(float, input().split()))
W2 = list(map(float, input().split()))
b2 = float(input())
dW1 = [[0.0] * n for _ in range(h)]
db1 = [0.0] * h
dW2 = [0.0] * h
db2 = 0.0
loss = 0.0
for x, y in data:
    z = [sum(w * xi for w, xi in zip(row, x)) + b for row, b in zip(W1, b1)]
    a = [max(0.0, v) for v in z]
    y_hat = sum(w * v for w, v in zip(W2, a)) + b2
    loss += (y_hat - y) ** 2 / N
    g = 2 * (y_hat - y) / N
    db2 += g
    for j in range(h):
        dW2[j] += g * a[j]
        if z[j] > 0:
            db1[j] += g * W2[j]
            for i in range(n):
                dW1[j][i] += g * W2[j] * x[i]


def fmt(values):
    return " ".join(f"{v + 0.0:.4f}" for v in values)


print(f"loss {loss:.4f}")
print("dW1: " + " | ".join(fmt(row) for row in dW1))
print("db1: " + fmt(db1))
print("dW2: " + fmt(dW2))
print(f"db2: {db2 + 0.0:.4f}")

Lesson: Training a multi-layer network →