Optimizers: momentum, RMSProp and Adam

Advanced

Introduction

Plain gradient descent uses one learning rate for every parameter and looks only at the current gradient. On real loss surfaces that is a problem: in a long narrow valley, the gradient points mostly across the valley, so the steps zigzag from side to side while progress along the valley floor is slow. Optimizers are better update rules. Nearly every network today is trained with Adam or SGD with momentum.

How it works

Mini-batch SGD estimates the gradient from a small random batch of examples. Each estimate is noisy but cheap, and the noise even helps escape poor regions.

Momentum keeps a velocity:

v ← β v + g, θ ← θ − η v

with β around 0.9. Gradients that keep pointing the same way add up and speed travel along the valley; gradients that flip sign cancel out and the zigzag is damped.

RMSProp gives each parameter its own effective learning rate. It keeps a running average s of squared gradients and divides the step by √s: parameters with large gradients take smaller steps, and parameters with small gradients take larger ones.

Adam combines the two. It keeps m, a running average of gradients (β₁ = 0.9), and s, a running average of squared gradients (β₂ = 0.999). Because both start at 0, early averages are too small, so Adam applies bias correction: m̂ = m / (1 − β₁ᵗ) and ŝ = s / (1 − β₂ᵗ) at step t. The update is

θ ← θ − η m̂ / (√ŝ + ε)

Since m̂ / √ŝ is about ±1 when the gradient is steady, each step is roughly η in size, whatever the gradient's scale.

Learning-rate schedules change η during training: step decay multiplies it by a factor every few epochs; exponential decay multiplies it by a factor every epoch; cosine decay lowers it smoothly along half a cosine wave to a minimum. Large steps early explore; small steps late settle.

Syntax

v = beta * v + g                 # momentum
theta -= rate * v

m = 0.9 * m + 0.1 * g            # Adam
s = 0.999 * s + 0.001 * g * g
m_hat, s_hat = m / (1 - 0.9 ** t), s / (1 - 0.999 ** t)
theta -= rate * m_hat / (math.sqrt(s_hat) + 1e-8)

Example

import math


def gradient(x, y):
    return 2 * x, 100 * y


def run(name, steps=60):
    x, y = 5.0, 1.0
    vx = vy = mx = my = sx = sy = 0.0
    for t in range(1, steps + 1):
        gx, gy = gradient(x, y)
        if name == "SGD":
            x, y = x - 0.019 * gx, y - 0.019 * gy
        elif name == "momentum":
            vx, vy = 0.9 * vx + gx, 0.9 * vy + gy
            x, y = x - 0.005 * vx, y - 0.005 * vy
        else:
            mx, my = 0.9 * mx + 0.1 * gx, 0.9 * my + 0.1 * gy
            sx, sy = 0.999 * sx + 0.001 * gx * gx, 0.999 * sy + 0.001 * gy * gy
            mhx, mhy = mx / (1 - 0.9 ** t), my / (1 - 0.9 ** t)
            shx, shy = sx / (1 - 0.999 ** t), sy / (1 - 0.999 ** t)
            x -= 0.5 * mhx / (math.sqrt(shx) + 1e-8)
            y -= 0.5 * mhy / (math.sqrt(shy) + 1e-8)
        if t in (10, 30, 60):
            print(f"{name:8} step {t:2}: x = {x:8.4f}, y = {y:8.4f}, f = {x * x + 50 * y * y:.5f}")


for name in ["SGD", "momentum", "Adam"]:
    run(name)
Open in compiler

Line by line

  1. The function x² + 50y² is a narrow valley: 50 times steeper in y than in x.
  2. SGD's learning rate must stay below 0.02, or the steep y direction diverges, so x shrinks slowly.
  3. Momentum builds up speed along the valley floor and ends about five times lower than SGD by step 60.
  4. Adam divides by each coordinate's own gradient size, so it makes quick progress in both directions from the start.
  5. The printed f values show how quickly each optimizer approaches the minimum at (0, 0).

Expected result

SGD      step 10: x =   3.3941, y =   0.3487, f = 17.59854
SGD      step 30: x =   1.5639, y =   0.0424, f = 2.53578
SGD      step 60: x =   0.4892, y =   0.0018, f = 0.23946
momentum step 10: x =   3.1054, y =   0.0877, f = 10.02826
momentum step 30: x =  -0.6403, y =  -0.1852, f = 2.12457
momentum step 60: x =   0.0198, y =   0.0318, f = 0.05081
Adam     step 10: x =   0.3812, y =   0.1803, f = 1.77144
Adam     step 30: x =  -0.1698, y =   0.1359, f = 0.95205
Adam     step 60: x =  -0.1733, y =   0.0111, f = 0.03623

In the real world

Adam and its variant AdamW (which handles weight decay separately) train most transformers and language models, usually with a warm-up followed by cosine decay. SGD with momentum remains popular for image models, where it often generalizes slightly better. Choosing and tuning the optimizer is a standard part of every training recipe.

Common mistakes

  • Using the same learning rate for SGD and Adam; Adam usually needs a much smaller one.
  • Forgetting bias correction, which makes Adam's first steps far too small.
  • Never lowering the learning rate, so the loss bounces around the minimum.
  • Comparing optimizers after a single run; results vary with the learning rate.

Best practices

  • Start with Adam at a learning rate around 0.001, or SGD with momentum 0.9.
  • Plot the loss with different learning rates before a long training run.
  • Use a schedule that lowers the learning rate towards the end.

Summary

Momentum accumulates a velocity that speeds consistent progress and damps zigzags. RMSProp and Adam scale each parameter's step by its recent gradient size, and Adam adds momentum and bias correction. Learning-rate schedules shrink the step size as training proceeds.

Quick check

1. Why does plain gradient descent zigzag in a narrow valley?
2. What does momentum keep?
3. Why does Adam apply bias correction?
4. What does cosine decay do?

Key takeaways

  • Momentum smooths the path and speeds progress along consistent directions.
  • Adam's steps are roughly the learning rate in size, whatever the scale of the gradient.
  • Lowering the learning rate late in training helps the model settle into a good minimum.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.