Probability and loss of the true token

Training scores a prediction by the probability it gave the token that really came next. Compute it.

What to build

A program that computes the probability, the cross-entropy loss and the top prediction for one step of a language model.

Requirements

  • Print P(TOKEN) P with 4 decimal places, for the actual token.
  • Print loss L, −ln P, with 4 decimal places.
  • Print top T, the most likely token (the first one on a tie).

Input

Line 1: the vocabulary. Line 2: one logit per token. Line 3: the actual next token.

Output

Three lines.

Constraints

1 to 1000 tokens; the actual token is in the vocabulary.

Example

Input:

cat dog sat
2 1 0
dog

Output:

P(dog) 0.2447
loss 1.4076
top cat

Why it works

The loss is small when the model gave the true token a high probability and grows quickly as that probability falls, which is the signal training uses to improve the model.

Examples

Example 1 · Input
cat dog sat
2 1 0
dog
Expected output
P(dog) 0.2447
loss 1.4076
top cat
Example 2 · Input
a b
0 0
a
Expected output
P(a) 0.5000
loss 0.6931
top a

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Turn the logits into probabilities with a stable softmax.

Hint 2

The loss is −ln of the probability of the actual token; the top token has the highest probability (the first on a tie).

Solution

Show one correct solution (try it yourself first)
import math

vocab = input().split()
logits = list(map(float, input().split()))
actual = input().strip()
top = max(logits)
exps = [math.exp(z - top) for z in logits]
probs = [e / sum(exps) for e in exps]
p = probs[vocab.index(actual)]
print(f"P({actual}) {p:.4f}")
print(f"loss {-math.log(p):.4f}")
print(f"top {vocab[probs.index(max(probs))]}")

Lesson: Next-token prediction →