Words to token IDs

A model reads numbers, not words. Encode a sentence with a word-level vocabulary.

What to build

A program that turns words into token IDs and counts the unknown ones.

Requirements

  • The vocabulary's IDs are its positions, from 0. The first entry is the unknown token.
  • Print the IDs of the text's words on one line.
  • Then print unknown U, the number of words not in the vocabulary.

Input

Line 1: the vocabulary. Line 2: the text.

Output

Two lines.

Constraints

1 to 10,000 words each.

Example

Input:

[UNK] the cat sat on mat
the cat sat on the hat

Output:

1 2 3 4 1 0
unknown 1

Why it works

A fixed vocabulary turns text into a sequence of numbers a model can compute with; any word outside it collapses into the unknown token, losing its meaning.

Examples

Example 1 · Input
[UNK] the cat sat on mat
the cat sat on the hat
Expected output
1 2 3 4 1 0
unknown 1
Example 2 · Input
[UNK] a b
b a
Expected output
2 1
unknown 0

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Build a dictionary from each vocabulary word to its position.

Hint 2

Words not in the dictionary get the ID of the first vocabulary entry, the unknown token.

Solution

Show one correct solution (try it yourself first)
vocab = input().split()
words = input().split()
ids = {w: i for i, w in enumerate(vocab)}
out = [ids.get(w, 0) for w in words]
print(*out)
print(f"unknown {sum(1 for w in words if w not in ids)}")

Lesson: Tokens and context windows →