A reward model is trained on pairs of answers: it should score the one people preferred higher.
What to build
A program that computes the agreement probability and loss for each preference pair, and their summary.
Requirements
- For pair K print
K: p P loss Lwith the probability P and the loss L to three decimal places. - Print
mean loss Mover all pairs, to three decimal places. - Print
agrees A/N, where A is the number of pairs with chosen score strictly above rejected score.
Input
Line 1: n. Then n lines, each with chosen rejected scores.
Output
n + 2 lines.
Constraints
1 ≤ n ≤ 50; scores between −10 and 10.
Example
Input:
2
2 -1
0 0
Output:
1: p 0.953 loss 0.049
2: p 0.500 loss 0.693
mean loss 0.371
agrees 1/2
Why it works
The sigmoid turns a score gap into the probability that the chosen answer wins, and the loss is small when the model is confident and right and large when it is confident and wrong, which is how a reward model learns from preferences.
Examples
2 2 -1 0 0
1: p 0.953 loss 0.049 2: p 0.500 loss 0.693 mean loss 0.371 agrees 1/2
1 -1 2
1: p 0.047 loss 3.049 mean loss 3.049 agrees 0/1
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
The chance that the reward model agrees with people is `1 / (1 + exp(-(chosen - rejected)))`.
Hint 2
The loss is the minus log of that chance; use `round(x, 12)` on results of `math.exp` and `math.log`.
Hint 3
A pair counts as agreed when the chosen score is higher than the rejected one.
Solution
Show one correct solution (try it yourself first)
import math
n = int(input())
total = 0.0
wins = 0
for i in range(1, n + 1):
c, r = map(float, input().split())
p = 1 / (1 + round(math.exp(-(c - r)), 12))
loss = round(-math.log(p), 12)
total += loss
wins += c > r
print(f"{i}: p {p:.3f} loss {loss:.3f}")
print(f"mean loss {total / n:.3f}")
print(f"agrees {wins}/{n}")