NLP: Evaluate a text classifier

Train Naive Bayes, run it on labelled test messages and measure how well it does.

What to build

A program that trains a classifier, lists its mistakes and reports accuracy, precision and recall.

Requirements

  • Train multinomial Naive Bayes with add-one smoothing on the training lines (label: text, lowercased, split on whitespace); test words never seen in training are ignored, and ties go to the alphabetically first label.
  • Classify every test line (also label: text). For each wrong prediction, in input order, print mistake: TEXT -> PREDICTED, where TEXT is the test text with surrounding spaces removed.
  • Then print accuracy A, and for each training label in alphabetical order label precision P recall R, all with three decimal places (0.000 when a divisor is 0).

Input

Line 1: n. Then n training lines. Then m and m test lines.

Output

The mistakes, then the accuracy line, then one line per label.

Constraints

n ≤ 500; 1 ≤ m ≤ 200.

Example

Input:

4
spam: win money now
spam: free money
ham: see you at lunch
ham: lunch money
4
spam: win free money
ham: lunch at noon
ham: free lunch
spam: see you money

Output:

mistake: see you money -> ham
accuracy 0.750
ham precision 0.667 recall 1.000
spam precision 1.000 recall 0.500

Why it works

Accuracy alone can hide a class the model never predicts; precision shows how trustworthy a predicted label is and recall how many true cases are found, so together they show where the classifier fails.

Examples

Example 1 · Input
4
spam: win money now
spam: free money
ham: see you at lunch
ham: lunch money
4
spam: win free money
ham: lunch at noon
ham: free lunch
spam: see you money
Expected output
mistake: see you money -> ham
accuracy 0.750
ham precision 0.667 recall 1.000
spam precision 1.000 recall 0.500
Example 2 · Input
2
pos: great fun
neg: awful dull
2
pos: fun
neg: dull film
Expected output
accuracy 1.000
neg precision 1.000 recall 1.000
pos precision 1.000 recall 1.000

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Train exactly as in the medium exercise, then classify each test message and compare with its true label.

Hint 2

precision = correct predictions of the class ÷ all predictions of the class; recall = correct predictions of the class ÷ test messages truly in the class. Use 0 when the divisor is 0.

Solution

Show one correct solution (try it yourself first)
import math

n = int(input())
docs, counts, totals, vocab = {}, {}, {}, []
for _ in range(n):
    label, text = input().split(":", 1)
    label = label.strip()
    docs[label] = docs.get(label, 0) + 1
    counts.setdefault(label, {})
    totals.setdefault(label, 0)
    for w in text.lower().split():
        counts[label][w] = counts[label].get(w, 0) + 1
        totals[label] += 1
        if w not in vocab:
            vocab.append(w)
labels = sorted(docs)


def predict(text):
    words = [w for w in text.lower().split() if w in vocab]
    scores = {}
    for label in labels:
        s = math.log(docs[label] / n)
        for w in words:
            s += math.log((counts[label].get(w, 0) + 1) / (totals[label] + len(vocab)))
        scores[label] = s
    return max(labels, key=lambda c: scores[c])


results = []
for _ in range(int(input())):
    actual, text = input().split(":", 1)
    actual, text = actual.strip(), text.strip()
    guess = predict(text)
    results.append((actual, guess))
    if guess != actual:
        print(f"mistake: {text} -> {guess}")
print(f"accuracy {sum(1 for a, g in results if a == g) / len(results):.3f}")
for label in labels:
    tp = sum(1 for a, g in results if a == label and g == label)
    predicted = sum(1 for a, g in results if g == label)
    actual = sum(1 for a, g in results if a == label)
    p = tp / predicted if predicted else 0.0
    r = tp / actual if actual else 0.0
    print(f"{label} precision {p:.3f} recall {r:.3f}")

Lesson: Naive Bayes text classification →