Training scores a prediction by the probability it gave the token that really came next. Compute it.
What to build
A program that computes the probability, the cross-entropy loss and the top prediction for one step of a language model.
Requirements
- Print
P(TOKEN) Pwith 4 decimal places, for the actual token. - Print
loss L, −ln P, with 4 decimal places. - Print
top T, the most likely token (the first one on a tie).
Input
Line 1: the vocabulary. Line 2: one logit per token. Line 3: the actual next token.
Output
Three lines.
Constraints
1 to 1000 tokens; the actual token is in the vocabulary.
Example
Input:
cat dog sat
2 1 0
dog
Output:
P(dog) 0.2447
loss 1.4076
top cat
Why it works
The loss is small when the model gave the true token a high probability and grows quickly as that probability falls, which is the signal training uses to improve the model.
Examples
cat dog sat 2 1 0 dog
P(dog) 0.2447 loss 1.4076 top cat
a b 0 0 a
P(a) 0.5000 loss 0.6931 top a
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Turn the logits into probabilities with a stable softmax.
Hint 2
The loss is −ln of the probability of the actual token; the top token has the highest probability (the first on a tie).
Solution
Show one correct solution (try it yourself first)
import math
vocab = input().split()
logits = list(map(float, input().split()))
actual = input().strip()
top = max(logits)
exps = [math.exp(z - top) for z in logits]
probs = [e / sum(exps) for e in exps]
p = probs[vocab.index(actual)]
print(f"P({actual}) {p:.4f}")
print(f"loss {-math.log(p):.4f}")
print(f"top {vocab[probs.index(max(probs))]}")