Write the character-level tokenizer from the lesson.
What to build
A program that splits a line into word tokens and punctuation tokens.
Requirements
- A word is a run of letters, digits and apostrophes (
'). Every other character that is not whitespace is a token on its own. - Print
tokens: N, then the tokens on one line separated by single spaces.
Input
One line of text.
Output
Two lines.
Constraints
The line has at most 1,000 characters and contains at least one token.
Example
Input:
Hello, world!
Output:
tokens: 4
Hello , world !
Why it works
Treating punctuation as separate tokens means "dogs." and "dogs" share the word "dogs", while keeping apostrophes inside words leaves contractions such as "can't" in one piece.
Examples
Example 1 · Input
Hello, world!
Expected output
tokens: 4 Hello , world !
Example 2 · Input
I can't stop (now).
Expected output
tokens: 7 I can't stop ( now ) .
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Walk through the text one character at a time, building up the current word.
Hint 2
Letters, digits (`ch.isalnum()`) and the apostrophe extend a word; any other character ends it, and if it is not a space it is a token by itself.
Solution
Show one correct solution (try it yourself first)
text = input()
tokens, word = [], ""
for ch in text + " ":
if ch.isalnum() or ch == "'":
word += ch
continue
if word:
tokens.append(word)
word = ""
if not ch.isspace():
tokens.append(ch)
print("tokens:", len(tokens))
print(" ".join(tokens))