Deep learning: Check a gradient numerically

Catch a wrong gradient before it silently ruins training.

What to build

A program that checks claimed gradients of a model's loss against finite differences.

Requirements

  • The model is ŷ = a x² + b x + c with mean squared error over the data. For each of a, b, c, compute the central-difference derivative with h = 0.0001 at the given parameters.
  • For each parameter print NAME: NUMERICAL ok or NAME: NUMERICAL mismatch (numerical value with four decimal places), where ok means a relative error below 0.00001.

Input

Line 1: x values. Line 2: y values. Line 3: a, b, c. Line 4: the claimed gradient for a, b, c.

Output

Three lines.

Constraints

At most 100 points; values below 1,000.

Example

Input:

1 2
1 4
0 0 0
-17 -9 -5

Output:

a: -17.0000 ok
b: -9.0000 ok
c: -5.0000 ok

Why it works

For a small h, the slope between two nearby points of the loss closely matches the true derivative, so a large disagreement means the claimed formula or code is wrong.

Examples

Example 1 · Input
1 2
1 4
0 0 0
-17 -9 -5
Expected output
a: -17.0000 ok
b: -9.0000 ok
c: -5.0000 ok
Example 2 · Input
1 2
1 4
0 0 0
-17 -9 -10
Expected output
a: -17.0000 ok
b: -9.0000 ok
c: -5.0000 mismatch

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Numerical derivative for one parameter: (L with the parameter + h − L with the parameter − h) / (2h), keeping the others fixed.

Hint 2

Relative error = |numerical − claimed| / max(1e-8, |numerical| + |claimed|).

Solution

Show one correct solution (try it yourself first)
xs = list(map(float, input().split()))
ys = list(map(float, input().split()))
params = list(map(float, input().split()))
claimed = list(map(float, input().split()))
H = 0.0001


def loss(p):
    a, b, c = p
    return sum((a * x * x + b * x + c - y) ** 2 for x, y in zip(xs, ys)) / len(xs)


for i, name in enumerate("abc"):
    up = params[:]
    down = params[:]
    up[i] += H
    down[i] -= H
    numerical = (loss(up) - loss(down)) / (2 * H)
    error = abs(numerical - claimed[i]) / max(1e-8, abs(numerical) + abs(claimed[i]))
    print(f"{name}: {numerical:.4f} {'ok' if error < 0.00001 else 'mismatch'}")

Lesson: Loss functions and gradient descent →