Deep learning: Full-batch training

Train a small network on a whole dataset per update.

What to build

A program that trains a one-hidden-layer ReLU network with full-batch gradient descent and reports the loss each epoch.

Requirements

  • The network maps n inputs to h ReLU units to one linear output; the loss is the mean of (ŷ − y)² over the N examples. ReLU's derivative is 1 for a positive input and 0 otherwise.
  • For each epoch print epoch E: loss L (four decimal places), the loss before that epoch's update.

Input

Line 1: n, h, N, the learning rate and the number of epochs. Then N lines: the inputs and the target. Then h rows of W₁, then b₁, then W₂, then b₂.

Output

One line per epoch.

Constraints

n, h ≤ 10; N ≤ 50; at most 100 epochs.

Example

Input:

1 1 2 0.05 2
1 2
2 4
1
0
1
0

Output:

epoch 1: loss 2.5000
epoch 2: loss 0.1495

Why it works

Averaging the gradient over every example gives the exact downhill direction of the mean loss, so each update improves the fit to the whole dataset at once.

Examples

Example 1 · Input
1 1 2 0.05 2
1 2
2 4
1
0
1
0
Expected output
epoch 1: loss 2.5000
epoch 2: loss 0.1495
Example 2 · Input
1 1 1 0.1 2
2 0
1
0
1
0
Expected output
epoch 1: loss 4.0000
epoch 2: loss 0.1600

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Each epoch: compute the mean loss over all examples, then the gradients averaged over the examples, then update once.

Hint 2

For example i, g = 2(ŷ − y) / N; the gradients are sums over the examples of the single-example formulas with that g.

Solution

Show one correct solution (try it yourself first)
n, h, N, rate, epochs = input().split()
n, h, N, rate, epochs = int(n), int(h), int(N), float(rate), int(epochs)
data = []
for _ in range(N):
    *x, y = map(float, input().split())
    data.append((x, y))
W1 = [list(map(float, input().split())) for _ in range(h)]
b1 = list(map(float, input().split()))
W2 = list(map(float, input().split()))
b2 = float(input())
for epoch in range(1, epochs + 1):
    dW1 = [[0.0] * n for _ in range(h)]
    db1 = [0.0] * h
    dW2 = [0.0] * h
    db2 = 0.0
    loss = 0.0
    for x, y in data:
        z = [sum(w * xi for w, xi in zip(row, x)) + b for row, b in zip(W1, b1)]
        a = [max(0.0, v) for v in z]
        y_hat = sum(w * v for w, v in zip(W2, a)) + b2
        loss += (y_hat - y) ** 2 / N
        g = 2 * (y_hat - y) / N
        for j in range(h):
            dW2[j] += g * a[j]
            if z[j] > 0:
                dz = g * W2[j]
                db1[j] += dz
                for i in range(n):
                    dW1[j][i] += dz * x[i]
        db2 += g
    print(f"epoch {epoch}: loss {loss:.4f}")
    for j in range(h):
        for i in range(n):
            W1[j][i] -= rate * dW1[j][i]
        b1[j] -= rate * db1[j]
        W2[j] -= rate * dW2[j]
    b2 -= rate * db2

Lesson: Training a multi-layer network →