NLP: Normalize and drop stopwords

Run the normalization steps of a pipeline in the right order.

What to build

A program that lowercases, cleans, maps numbers and removes stopwords from a line of text.

Requirements

  • Line 1 holds the stopwords (lowercase). For each whitespace-separated piece of line 2: lowercase it, strip the characters . , ! ? ; : " ' ( ) - from both ends, and skip it if it becomes empty.
  • A cleaned word made only of digits becomes NUM. Then drop the word if it is a stopword.
  • Print the kept words separated by spaces, then kept K of N, where N counts the cleaned non-empty words before stopwords were removed.

Input

Line 1: stopwords. Line 2: the text.

Output

Two lines.

Constraints

At most 1,000 characters per line; at least one word is kept.

Example

Input:

the a is of
The cat is on the mat.

Output:

cat on mat
kept 3 of 6

Why it works

Cleaning before comparing makes "The" and "the," the same word, mapping numbers to one symbol stops every number becoming its own vocabulary entry, and dropping stopwords leaves the words that carry the topic.

Examples

Example 1 · Input
the a is of
The cat is on the mat.
Expected output
cat on mat
kept 3 of 6
Example 2 · Input
and to
I ran 5 km, and walked 10!
Expected output
i ran NUM km walked NUM
kept 6 of 7

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

For each whitespace piece: `raw.lower().strip(".,!?;:\"'()-")`; skip it if nothing is left.

Hint 2

`word.isdigit()` tells you whether the cleaned word is a number.

Solution

Show one correct solution (try it yourself first)
stop = set(input().split())
kept, total = [], 0
for raw in input().split():
    word = raw.lower().strip(".,!?;:\"'()-")
    if not word:
        continue
    total += 1
    if word.isdigit():
        word = "NUM"
    if word not in stop:
        kept.append(word)
print(" ".join(kept))
print(f"kept {len(kept)} of {total}")

Lesson: Tokenization and normalization →