Deep learning: L2 penalty and weight decay

Add L2 regularization to a loss and to a gradient step.

What to build

A program that computes an L2-regularized loss and performs one weight-decay update.

Requirements

  • Inputs: the data loss, λ, the learning rate, the weights and their data-loss gradients.
  • Print penalty P, total T and weights: followed by the updated weights, all with four decimal places.

Input

Line 1: the data loss, λ and the learning rate. Line 2: the weights. Line 3: their gradients.

Output

Three lines.

Constraints

At most 1,000 weights.

Example

Input:

1.5 0.1 0.5
1 -2
0 0

Output:

penalty 0.5000
total 2.0000
weights: 0.9000 -1.8000

Why it works

The penalty's gradient 2λw points straight towards zero, so every step pulls each weight a little towards zero in addition to fitting the data.

Examples

Example 1 · Input
1.5 0.1 0.5
1 -2
0 0
Expected output
penalty 0.5000
total 2.0000
weights: 0.9000 -1.8000
Example 2 · Input
0.2 0 0.1
3 4
1 -1
Expected output
penalty 0.0000
total 0.2000
weights: 2.9000 4.1000

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

The penalty is λ × Σ w², and the total loss is the data loss plus the penalty.

Hint 2

Each new weight is w − rate × (g + 2λw).

Solution

Show one correct solution (try it yourself first)
loss, lam, rate = map(float, input().split())
ws = list(map(float, input().split()))
gs = list(map(float, input().split()))
penalty = lam * sum(w * w for w in ws)
print(f"penalty {penalty:.4f}")
print(f"total {loss + penalty:.4f}")
print("weights: " + " ".join(f"{w - rate * (g + 2 * lam * w) + 0.0:.4f}" for w, g in zip(ws, gs)))

Lesson: Overfitting and regularization →