Compute the exact gradients that a full-batch update uses, for every parameter.
What to build
A program that computes the mean loss and all parameter gradients of a one-hidden-layer ReLU network over a batch.
Requirements
- The network and loss are those of the previous exercise.
- Print
loss L, thendW1:(rows separated by|),db1:,dW2:anddb2:, all with four decimal places.
Input
Line 1: n, h and N. Then N lines: the inputs and the target. Then h rows of W₁, then b₁, then W₂, then b₂.
Output
Five lines.
Constraints
n, h ≤ 10; N ≤ 50.
Example
Input:
1 1 2
1 2
2 4
1
0
1
0
Output:
loss 2.5000
dW1: -5.0000
db1: -3.0000
dW2: -5.0000
db2: -3.0000
Why it works
The mean loss is an average, so its gradient is the average of the single-example gradients, each found by backpropagation.
Examples
1 1 2 1 2 2 4 1 0 1 0
loss 2.5000 dW1: -5.0000 db1: -3.0000 dW2: -5.0000 db2: -3.0000
2 2 1 1 2 3 1 0 0 1 0 -3 1 1 0
loss 4.0000 dW1: -4.0000 -8.0000 | 0.0000 0.0000 db1: -4.0000 0.0000 dW2: -4.0000 0.0000 db2: -4.0000
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Run the forward pass for every example, keeping z₁ and the hidden activations, and add each example's contribution to the gradient sums.
Hint 2
With the mean squared error, each example's output gradient is g = 2(ŷ − y) / N; everything else follows the single-example rules.
Solution
Show one correct solution (try it yourself first)
n, h, N = map(int, input().split())
data = []
for _ in range(N):
*x, y = map(float, input().split())
data.append((x, y))
W1 = [list(map(float, input().split())) for _ in range(h)]
b1 = list(map(float, input().split()))
W2 = list(map(float, input().split()))
b2 = float(input())
dW1 = [[0.0] * n for _ in range(h)]
db1 = [0.0] * h
dW2 = [0.0] * h
db2 = 0.0
loss = 0.0
for x, y in data:
z = [sum(w * xi for w, xi in zip(row, x)) + b for row, b in zip(W1, b1)]
a = [max(0.0, v) for v in z]
y_hat = sum(w * v for w, v in zip(W2, a)) + b2
loss += (y_hat - y) ** 2 / N
g = 2 * (y_hat - y) / N
db2 += g
for j in range(h):
dW2[j] += g * a[j]
if z[j] > 0:
db1[j] += g * W2[j]
for i in range(n):
dW1[j][i] += g * W2[j] * x[i]
def fmt(values):
return " ".join(f"{v + 0.0:.4f}" for v in values)
print(f"loss {loss:.4f}")
print("dW1: " + " | ".join(fmt(row) for row in dW1))
print("db1: " + fmt(db1))
print("dW2: " + fmt(dW2))
print(f"db2: {db2 + 0.0:.4f}")