Choose a decision threshold

A probability becomes a decision only once a threshold is chosen. Try several and compare them.

What to build

A program that scores a classifier at several thresholds and picks the one with the best F1.

Requirements

  • For each threshold t (in the given order), predict 1 when score ≥ t.
  • Print T: precision P recall R F1 F, with T as in the input and the scores with 3 decimal places.
  • Then print best threshold T for the highest F1; on a tie, the earlier threshold.

Input

Line 1: the actual labels (0 or 1). Line 2: the scores (probabilities of class 1). Line 3: the thresholds.

Output

One line per threshold, then the best.

Constraints

1 to 1000 examples; 1 to 20 thresholds.

Example

Input:

1 1 0 1 0 0
0.9 0.8 0.7 0.4 0.3 0.1
0.25 0.5 0.75

Output:

0.25: precision 0.600 recall 1.000 F1 0.750
0.5: precision 0.667 recall 0.667 F1 0.667
0.75: precision 1.000 recall 0.667 F1 0.800
best threshold 0.75

Why it works

Lowering the threshold flags more examples, which can only raise recall and usually lowers precision; F1 finds the threshold where the two are best balanced.

Examples

Example 1 · Input
1 1 0 1 0 0
0.9 0.8 0.7 0.4 0.3 0.1
0.25 0.5 0.75
Expected output
0.25: precision 0.600 recall 1.000 F1 0.750
0.5: precision 0.667 recall 0.667 F1 0.667
0.75: precision 1.000 recall 0.667 F1 0.800
best threshold 0.75
Example 2 · Input
0 1
0.2 0.6
0.5 0.9
Expected output
0.5: precision 1.000 recall 1.000 F1 1.000
0.9: precision 0.000 recall 0.000 F1 0.000
best threshold 0.5

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

For each threshold, predict 1 when the score is at least the threshold, then count TP, FP and FN.

Hint 2

Precision, recall or F1 is 0 when its formula would divide by zero.

Hint 3

Keep the first threshold with the highest F1.

Solution

Show one correct solution (try it yourself first)
labels = list(map(int, input().split()))
scores = list(map(float, input().split()))
thresholds = input().split()
best = None
for t in thresholds:
    predicted = [1 if s >= float(t) else 0 for s in scores]
    tp = sum(1 for y, p in zip(labels, predicted) if y == 1 and p == 1)
    fp = sum(1 for y, p in zip(labels, predicted) if y == 0 and p == 1)
    fn = sum(1 for y, p in zip(labels, predicted) if y == 1 and p == 0)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    print(f"{t}: precision {precision:.3f} recall {recall:.3f} F1 {f1:.3f}")
    if best is None or f1 > best[1]:
        best = (t, f1)
print(f"best threshold {best[0]}")

Lesson: Measuring a classifier →