Introduction
In a bag of words, "the" has the biggest count in almost every English document, yet it says nothing about what the document is about. The words that matter are those that are frequent in this document but rare in the others: "mat" in a story about a mat, "goal" in a football report. TF-IDF (term frequency × inverse document frequency) captures exactly this idea in one number per word. It has been the standard weighting for search engines and document classification for decades and is still a strong baseline today.
How it works
For a word t, a document d and a collection of N documents:
- Term frequency: tf(t, d) = count of t in d ÷ number of tokens in d. Dividing by the length stops long documents from winning just by being long.
- Document frequency: df(t) = number of documents that contain t at least once.
- Inverse document frequency: idf(t) = ln(N ÷ df(t)). A word in every document gets ln(1) = 0; a word in only one document gets the largest value, ln N.
- TF-IDF: tfidf(t, d) = tf(t, d) × idf(t).
Example with N = 3 documents: "the" is in all three, so its idf is ln(3/3) = 0 and its weight is 0 everywhere, however often it occurs. "mat" is in one document, so its idf is ln 3 ≈ 1.099. In "the cat sat on the mat" (6 tokens), mat's weight is 1/6 × 1.099 ≈ 0.183, higher than "cat" (in two documents, idf ln 1.5 ≈ 0.405, weight ≈ 0.068).
Variants you will meet:
- Raw counts or log counts (1 + ln count) instead of tf as a fraction.
- Smoothed idf: ln((1 + N) ÷ (1 + df)) + 1, as in scikit-learn, which never divides by zero and never gives exactly 0.
- L2 normalization: dividing each document's vector by its length, so documents of different sizes compare fairly.
The numbers differ between variants, but the idea is the same: frequent here, rare elsewhere.
Example
import math
docs = [
"the cat sat on the mat",
"the dog sat on the log",
"the cat chased the dog",
]
tokenized = [d.split() for d in docs]
N = len(docs)
df = {}
for doc in tokenized:
for word in sorted(set(doc)):
df[word] = df.get(word, 0) + 1
idf = {w: round(math.log(N / df[w]), 12) for w in sorted(df)}
for w in idf:
print(f"{w:7} df={df[w]} idf={idf[w]:.3f}")
for i, doc in enumerate(tokenized, 1):
scores = {}
for w in doc:
scores[w] = doc.count(w) / len(doc) * idf[w]
best = sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))[:2]
print(f"doc {i} keywords:", [(w, round(s, 3)) for w, s in best])Line by line
dfcounts, for each word, how many documents contain it. Iterating oversorted(set(doc))counts each word once per document, in a fixed order.idfuses ln(N / df). Theround(..., 12)makes the stored value identical on every computer, because maths libraries can differ in the last digit.- "the" appears in all three documents, so its idf is 0.
- For each document, a word's score is its term frequency times its idf.
- Sorting by
(-score, word)gives the top keywords, with ties broken alphabetically: "mat", "log" and "chased" stand out, the words unique to their documents.
Expected result
cat df=2 idf=0.405
chased df=1 idf=1.099
dog df=2 idf=0.405
log df=1 idf=1.099
mat df=1 idf=1.099
on df=2 idf=0.405
sat df=2 idf=0.405
the df=3 idf=0.000
doc 1 keywords: [('mat', 0.183), ('cat', 0.068)]
doc 2 keywords: [('log', 0.183), ('dog', 0.068)]
doc 3 keywords: [('chased', 0.22), ('cat', 0.081)]
In the real world
TF-IDF was the core of search engines for decades and still ranks results inside many search tools; its refined cousin BM25 is the default ranking function in Lucene and Elasticsearch. With scikit-learn (shown, not run: it is not available in StepVioz's browser Python; install it with pip install scikit-learn):
from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["the cat sat on the mat", "the dog sat on the log", "the cat chased the dog"]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
print(X.toarray().round(3))
Its default uses smoothed idf and L2 normalization, so its numbers differ from this lesson's, but the ranking of keywords is similar. TF-IDF vectors also feed classifiers such as logistic regression, a strong baseline for text classification.
Common mistakes
- Using the wrong log base or variant and then comparing numbers with another tool's.
- Computing idf on the test documents: like the vocabulary, idf must come from the training collection.
- Dividing by zero for a word that is in no document, such as a query word; skip such words or use smoothed idf.
- Forgetting normalization when comparing documents of very different lengths.
Best practices
- Compute document frequencies once over the collection and reuse them.
- Remove or down-weight stopwords; TF-IDF already does most of this for you.
- Write down which variant you use (tf, idf and normalization) in your code.
- Check a few documents' top-weighted words by eye: they should look like keywords.
Summary
TF-IDF weights a word by its frequency in a document times the log of how rare it is across the collection. Words in every document get weight 0, and words concentrated in one document get high weights, so the top-weighted words are a document's keywords. Libraries use variants (smoothed idf, normalization), but the principle is the same.
Quick check
Key takeaways
- Raw counts reward common words such as "the"; IDF scales them down according to how many documents contain them.
- A word with a high TF-IDF weight is characteristic of its document, which makes it a good keyword.
- Libraries use variants of the formula, so always check which one before comparing numbers.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.