Deep learning: Cross-entropy and its gradient

Compute the loss and gradient at the end of every classifier.

What to build

A program that computes the cross-entropy loss of logits for a true class and the gradient of the loss with respect to the logits.

Requirements

  • Use softmax with the largest logit subtracted. The loss is −log p_y; the gradient is p − one-hot(y).
  • Print loss L, then grad: and the gradient values, all with four decimal places.

Input

Line 1: the logits. Line 2: the true class (from 0).

Output

Two lines.

Constraints

At most 100 logits.

Example

Input:

0 0
0

Output:

loss 0.6931
grad: -0.5000 0.5000

Why it works

Raising the true class's logit lowers the loss most when its probability is far from 1, and the gradient p − one-hot says exactly how much each logit should move.

Examples

Example 1 · Input
0 0
0
Expected output
loss 0.6931
grad: -0.5000 0.5000
Example 2 · Input
1 2 3
2
Expected output
loss 0.4076
grad: 0.0900 0.2447 -0.3348

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Compute stable softmax probabilities first.

Hint 2

The loss is −log p[class]; the gradient is p with 1 subtracted at the true class.

Solution

Show one correct solution (try it yourself first)
import math

zs = list(map(float, input().split()))
y = int(input())
top = max(zs)
exps = [math.exp(z - top) for z in zs]
p = [e / sum(exps) for e in exps]
print(f"loss {-math.log(p[y]):.4f}")
print("grad: " + " ".join(f"{pk - (1 if k == y else 0):.4f}" for k, pk in enumerate(p)))

Lesson: Softmax and multi-class classification →