Compute every part of the TF-IDF formula for the words of one document.
What to build
A program that prints tf, idf and tf-idf for each distinct word of a chosen document.
Requirements
- Lowercase the documents and split them on whitespace. Use tf = count ÷ length, idf = ln(N ÷ df) with the natural logarithm, and tf-idf = tf × idf.
- For each distinct word of document i, in order of first appearance, print
word tf idf tfidf, each number with four decimal places.
Input
Line 1: N. Then N documents. Last line: i, the number of the document to analyse (from 1).
Output
One line per distinct word of document i.
Constraints
N ≤ 100.
Example
Input:
2
the cat sat
the dog
1
Output:
the 0.3333 0.0000 0.0000
cat 0.3333 0.6931 0.2310
sat 0.3333 0.6931 0.2310
Why it works
tf rewards words used often in the document, idf rewards words found in few documents, and their product is high only for words that are both.
Examples
2 the cat sat the dog 1
the 0.3333 0.0000 0.0000 cat 0.3333 0.6931 0.2310 sat 0.3333 0.6931 0.2310
4 a b a c a d b b c e 4
b 0.5000 0.6931 0.3466 c 0.2500 0.6931 0.1733 e 0.2500 1.3863 0.3466
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
tf = count in the document ÷ the document's length; df = number of documents containing the word; idf = `math.log(n / df)`.
Hint 2
Keep the distinct words of the chosen document in order of first appearance.
Solution
Show one correct solution (try it yourself first)
import math
n = int(input())
docs = [input().lower().split() for _ in range(n)]
doc = docs[int(input()) - 1]
seen = []
for w in doc:
if w not in seen:
seen.append(w)
for w in seen:
tf = doc.count(w) / len(doc)
df = sum(1 for d in docs if w in d)
idf = math.log(n / df)
print(f"{w} {tf:.4f} {idf:.4f} {tf * idf:.4f}")