Deep learning: Learning-rate schedules

Compute how the learning rate changes over training.

What to build

A program that prints the learning rate for every epoch under a schedule.

Requirements

  • Schedules: step K F (multiply by F every K epochs), exponential D (multiply by D every epoch), cosine MIN (decrease along half a cosine from rate₀ at epoch 1 to MIN at epoch E). Epochs are numbered from 1.
  • Print epoch e: rate with four decimal places for every epoch.

Input

One line: rate₀, E, the schedule name and its parameters.

Output

E lines.

Constraints

2 ≤ E ≤ 1,000.

Example

Input:

0.1 5 step 2 0.5

Output:

epoch 1: 0.1000
epoch 2: 0.1000
epoch 3: 0.0500
epoch 4: 0.0500
epoch 5: 0.0250

Why it works

Large steps early make quick progress, and smaller steps later let the parameters settle near a minimum instead of bouncing around it.

Examples

Example 1 · Input
0.1 5 step 2 0.5
Expected output
epoch 1: 0.1000
epoch 2: 0.1000
epoch 3: 0.0500
epoch 4: 0.0500
epoch 5: 0.0250
Example 2 · Input
1 4 exponential 0.5
Expected output
epoch 1: 1.0000
epoch 2: 0.5000
epoch 3: 0.2500
epoch 4: 0.1250

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Step decay: rate₀ × F^((e − 1) // K). Exponential: rate₀ × D^(e − 1).

Hint 2

Cosine: MIN + 0.5 × (rate₀ − MIN) × (1 + cos(π (e − 1) / (E − 1))), using math.cos and math.pi.

Solution

Show one correct solution (try it yourself first)
import math

parts = input().split()
rate0, E, kind = float(parts[0]), int(parts[1]), parts[2]
for e in range(1, E + 1):
    if kind == "step":
        rate = rate0 * float(parts[4]) ** ((e - 1) // int(parts[3]))
    elif kind == "exponential":
        rate = rate0 * float(parts[3]) ** (e - 1)
    else:
        low = float(parts[3])
        rate = low + 0.5 * (rate0 - low) * (1 + math.cos(math.pi * (e - 1) / (E - 1)))
    print(f"epoch {e}: {rate + 0.0:.4f}")

Lesson: Optimizers: momentum, RMSProp and Adam →