Score preference pairs with a reward model

A reward model is trained on pairs of answers: it should score the one people preferred higher.

What to build

A program that computes the agreement probability and loss for each preference pair, and their summary.

Requirements

  • For pair K print K: p P loss L with the probability P and the loss L to three decimal places.
  • Print mean loss M over all pairs, to three decimal places.
  • Print agrees A/N, where A is the number of pairs with chosen score strictly above rejected score.

Input

Line 1: n. Then n lines, each with chosen rejected scores.

Output

n + 2 lines.

Constraints

1 ≤ n ≤ 50; scores between −10 and 10.

Example

Input:

2
2 -1
0 0

Output:

1: p 0.953 loss 0.049
2: p 0.500 loss 0.693
mean loss 0.371
agrees 1/2

Why it works

The sigmoid turns a score gap into the probability that the chosen answer wins, and the loss is small when the model is confident and right and large when it is confident and wrong, which is how a reward model learns from preferences.

Examples

Example 1 · Input
2
2 -1
0 0
Expected output
1: p 0.953 loss 0.049
2: p 0.500 loss 0.693
mean loss 0.371
agrees 1/2
Example 2 · Input
1
-1 2
Expected output
1: p 0.047 loss 3.049
mean loss 3.049
agrees 0/1

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

The chance that the reward model agrees with people is `1 / (1 + exp(-(chosen - rejected)))`.

Hint 2

The loss is the minus log of that chance; use `round(x, 12)` on results of `math.exp` and `math.log`.

Hint 3

A pair counts as agreed when the chosen score is higher than the rejected one.

Solution

Show one correct solution (try it yourself first)
import math

n = int(input())
total = 0.0
wins = 0
for i in range(1, n + 1):
    c, r = map(float, input().split())
    p = 1 / (1 + round(math.exp(-(c - r)), 12))
    loss = round(-math.log(p), 12)
    total += loss
    wins += c > r
    print(f"{i}: p {p:.3f} loss {loss:.3f}")
print(f"mean loss {total / n:.3f}")
print(f"agrees {wins}/{n}")

Lesson: Fine-tuning and alignment →