Naive Bayes text classification

Intermediate

Introduction

Is an email spam? Is a support ticket about billing or about a bug? Which language is this text in? Text classification assigns a label to a document, and Naive Bayes is the classic first method. It trains in one pass of counting, needs very little data, is easy to explain, and was the engine of the first successful spam filters. It is still a strong baseline that more complex models must beat.

How it works

Bayes' rule says that the probability of class c given a document d is proportional to the prior times the likelihood:

P(c given d) ∝ P(c) × P(d given c)

The naive assumption is that, once the class is known, each word is independent of the others. Then the likelihood of the document is a product over its words:

P(d given c) = P(w₁ given c) × P(w₂ given c) × …

Both parts come from counting the training data:

  • Prior: P(c) = number of documents in class c ÷ number of documents.
  • Likelihood (multinomial model): P(w given c) = count of w in class c ÷ total words in class c.

A word that never appeared in the spam training texts would get probability 0, and one zero wipes out the whole product. Add-one (Laplace) smoothing fixes this:

P(w given c) = (count of w in c + 1) ÷ (total words in c + V)

where V is the vocabulary size of the training data. Words never seen in training at all are simply ignored.

Multiplying many probabilities below 1 quickly gives numbers too small for a computer (underflow). Since log(a × b) = log a + log b, classifiers add log probabilities instead:

score(c) = log P(c) + Σ log P(w given c)

and choose the class with the highest score. To turn the scores back into probabilities, exponentiate their differences: for two classes, P(spam) = 1 ÷ (1 + e^(score(ham) − score(spam))).

Example

import math

train = [
    ("spam", "win money now"),
    ("spam", "win a free prize now"),
    ("ham", "meeting at noon"),
    ("ham", "lunch at noon with the team"),
]
classes = ["ham", "spam"]
docs = {c: 0 for c in classes}
totals = {c: 0 for c in classes}
counts = {c: {} for c in classes}
vocab = []
for label, text in train:
    docs[label] += 1
    for w in text.split():
        counts[label][w] = counts[label].get(w, 0) + 1
        totals[label] += 1
        if w not in vocab:
            vocab.append(w)
V = len(vocab)
print("vocabulary size:", V, " tokens per class:", totals)


def log_score(label, words):
    score = round(math.log(docs[label] / len(train)), 12)
    for w in words:
        if w in vocab:
            p = (counts[label].get(w, 0) + 1) / (totals[label] + V)
            score += round(math.log(p), 12)
    return score


for message in ["free lunch now", "lunch at noon"]:
    words = message.split()
    scores = {c: log_score(c, words) for c in classes}
    best = max(classes, key=lambda c: scores[c])
    p_spam = 1 / (1 + round(math.exp(scores["ham"] - scores["spam"]), 12))
    print(f"{message!r}: ham {scores['ham']:.3f}, spam {scores['spam']:.3f} -> {best} (P(spam) = {p_spam:.3f})")
Open in compiler

Line by line

  1. Training counts documents per class (the priors), words per class and the total number of words per class.
  2. The vocabulary has 13 distinct words; spam has 8 word tokens, ham 9.
  3. log_score starts from the log prior, ln 0.5, and adds the log of each smoothed likelihood. Logs are rounded to 12 places so every computer stores the same digits.
  4. "free lunch now": "free" and "now" are spam words, "lunch" a ham word. Spam has the higher (less negative) score, -8.035 against -9.273.
  5. "lunch at noon" only contains ham words, so ham wins clearly, and P(spam) is only about 6%.

Expected result

vocabulary size: 13  tokens per class: {'ham': 9, 'spam': 8}
'free lunch now': ham -9.273, spam -8.035 -> spam (P(spam) = 0.775)
'lunch at noon': ham -7.076, spam -9.827 -> ham (P(spam) = 0.060)

In the real world

scikit-learn implements this model as MultinomialNB (shown, not run: it is not available in StepVioz's browser Python):

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ["win money now", "win a free prize now", "meeting at noon", "lunch at noon with the team"]
labels = ["spam", "spam", "ham", "ham"]
vectorizer = CountVectorizer()
model = MultinomialNB(alpha=1.0).fit(vectorizer.fit_transform(texts), labels)
print(model.predict(vectorizer.transform(["free lunch now"])))

alpha=1.0 is add-one smoothing. Naive Bayes is still used where speed and simplicity matter: spam filtering, language identification, routing messages and as a quick baseline. Logistic regression and fine-tuned transformers usually reach higher accuracy, because they do not assume that words are independent.

Common mistakes

  • Forgetting smoothing, so any new word makes a class impossible.
  • Multiplying raw probabilities for long documents until they underflow to 0.
  • Using different tokenization for training and prediction.
  • Ignoring class imbalance: with 99% ham in training, the prior alone pushes predictions towards ham, which may or may not be what you want.

Best practices

  • Work in log space from the start.
  • Fit the vocabulary and counts on the training set only, and evaluate on separate test data.
  • Try binary counts (a word counts once per document) as well as full counts; for short texts they often work better.
  • Look at the words with the largest likelihood ratios between classes to understand what the model learned.

Summary

Naive Bayes classifies a document by combining a class prior with the likelihood of each word in that class, assuming the words are independent given the class. Priors and likelihoods are counts, add-one smoothing keeps unseen words from giving zero probability, and summing log probabilities avoids underflow.

Quick check

1. What is the 'naive' assumption in Naive Bayes?
2. Why is add-one smoothing used?
3. Why add log probabilities instead of multiplying probabilities?
4. A class has 30 of the 120 training documents. What is its prior?

Key takeaways

  • Naive Bayes picks the class that maximizes prior × product of word likelihoods.
  • Without smoothing, one unseen word makes a whole class impossible.
  • Summing log probabilities keeps the computation accurate for long documents.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.