NLP: Precision, recall and F1

Compute the evaluation report that every classifier needs.

What to build

A program that prints precision, recall and F1 for each class, the macro F1 and the accuracy.

Requirements

  • Each input pair is actual predicted. The classes are every label that appears as actual or predicted, in alphabetical order.
  • For each class print class: precision P recall R f1 F. Precision is 0 when the class is never predicted, recall is 0 when it never occurs, and F1 is 0 when P + R = 0.
  • Then print macro f1 M (the average F1 over the classes) and accuracy A. All numbers have three decimal places.

Input

Line 1: n. Then n lines actual predicted.

Output

One line per class, then two lines.

Constraints

1 ≤ n ≤ 10,000.

Example

Input:

6
pos pos
neg neg
pos pos
neg neg
neg neutral
neg neg

Output:

neg: precision 1.000 recall 0.750 f1 0.857
neutral: precision 0.000 recall 0.000 f1 0.000
pos: precision 1.000 recall 1.000 f1 1.000
macro f1 0.619
accuracy 0.833

Why it works

Per-class precision and recall show which kind of mistake a model makes, F1 combines them, and the macro average gives a small class the same weight as a large one, unlike accuracy.

Examples

Example 1 · Input
6
pos pos
neg neg
pos pos
neg neg
neg neutral
neg neg
Expected output
neg: precision 1.000 recall 0.750 f1 0.857
neutral: precision 0.000 recall 0.000 f1 0.000
pos: precision 1.000 recall 1.000 f1 1.000
macro f1 0.619
accuracy 0.833
Example 2 · Input
4
spam ham
spam spam
ham ham
ham spam
Expected output
ham: precision 0.500 recall 0.500 f1 0.500
spam: precision 0.500 recall 0.500 f1 0.500
macro f1 0.500
accuracy 0.500

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

For class c: true positives = pairs with actual c and predicted c; precision divides them by all predictions of c, recall by all actual c.

Hint 2

F1 = 2PR / (P + R), and 0 when P + R is 0; macro F1 is the plain average of the classes' F1.

Solution

Show one correct solution (try it yourself first)
n = int(input())
pairs = [input().split() for _ in range(n)]
labels = sorted({a for a, _ in pairs} | {p for _, p in pairs})
f1s = []
for c in labels:
    tp = sum(1 for a, p in pairs if a == c and p == c)
    predicted = sum(1 for _, p in pairs if p == c)
    actual = sum(1 for a, _ in pairs if a == c)
    precision = tp / predicted if predicted else 0.0
    recall = tp / actual if actual else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    f1s.append(f1)
    print(f"{c}: precision {precision:.3f} recall {recall:.3f} f1 {f1:.3f}")
print(f"macro f1 {sum(f1s) / len(f1s):.3f}")
print(f"accuracy {sum(1 for a, p in pairs if a == p) / n:.3f}")

Lesson: Sentiment analysis →