NLP: Split words from punctuation

Write the character-level tokenizer from the lesson.

What to build

A program that splits a line into word tokens and punctuation tokens.

Requirements

  • A word is a run of letters, digits and apostrophes ('). Every other character that is not whitespace is a token on its own.
  • Print tokens: N, then the tokens on one line separated by single spaces.

Input

One line of text.

Output

Two lines.

Constraints

The line has at most 1,000 characters and contains at least one token.

Example

Input:

Hello, world!

Output:

tokens: 4
Hello , world !

Why it works

Treating punctuation as separate tokens means "dogs." and "dogs" share the word "dogs", while keeping apostrophes inside words leaves contractions such as "can't" in one piece.

Examples

Example 1 · Input
Hello, world!
Expected output
tokens: 4
Hello , world !
Example 2 · Input
I can't stop (now).
Expected output
tokens: 7
I can't stop ( now ) .

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Walk through the text one character at a time, building up the current word.

Hint 2

Letters, digits (`ch.isalnum()`) and the apostrophe extend a word; any other character ends it, and if it is not a space it is a token by itself.

Solution

Show one correct solution (try it yourself first)
text = input()
tokens, word = [], ""
for ch in text + " ":
    if ch.isalnum() or ch == "'":
        word += ch
        continue
    if word:
        tokens.append(word)
        word = ""
    if not ch.isspace():
        tokens.append(ch)
print("tokens:", len(tokens))
print(" ".join(tokens))

Lesson: Tokenization and normalization →