Lex a multi-line program and record where every token starts.
What to build
A program that tokenizes several lines and prints every token with its position.
Requirements
- Rules: STRING (
"…"with no quote inside), NUMBER, NAME/KEYWORD (if,else,while,print,def,return), OP (==,!=,<=,>=, then one of+ - * / = < > ( ) , :), # comments, whitespace. - Print
line:column KIND lexeme. On an unexpected character printline:column errorand stop the whole program.
Input
Line 1: n. Then n lines of code.
Output
One line per token, possibly ending with an error line.
Constraints
n ≤ 10; lines have at most 80 characters.
Example
Input:
2
def area(r):
return 3.14 * r * r
Output:
1:1 KEYWORD def
1:5 NAME area
1:9 OP (
1:10 NAME r
1:11 OP )
1:12 OP :
2:5 KEYWORD return
2:12 NUMBER 3.14
2:17 OP *
2:19 NAME r
2:21 OP *
2:23 NAME r
Why it works
Lexing line by line and keeping each match's start position gives exact locations, which every later error message depends on.
Examples
Example 1 · Input
2
def area(r):
return 3.14 * r * r
Expected output
1:1 KEYWORD def 1:5 NAME area 1:9 OP ( 1:10 NAME r 1:11 OP ) 1:12 OP : 2:5 KEYWORD return 2:12 NUMBER 3.14 2:17 OP * 2:19 NAME r 2:21 OP * 2:23 NAME r
Example 2 · Input
1
print("hi", x)
Expected output
1:1 KEYWORD print 1:6 OP ( 1:7 STRING "hi" 1:11 OP , 1:13 NAME x 1:14 OP )
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Run the master pattern on each line separately, counting lines from 1, and use m.start() + 1 for the column.
Hint 2
Put the STRING rule (double quotes, no newline inside) before the operators.
Solution
Show one correct solution (try it yourself first)
import re
KEYWORDS = {"if", "else", "while", "print", "def", "return"}
SPEC = [
("STRING", r'"[^"\n]*"'),
("NUMBER", r"\d+(\.\d+)?"),
("NAME", r"[A-Za-z_]\w*"),
("OP", r"==|!=|<=|>=|[-+*/=<>(),:]"),
("COMMENT", r"#.*"),
("SKIP", r"\s+"),
("MISMATCH", r"."),
]
MASTER = re.compile("|".join(f"(?P<{n}>{p})" for n, p in SPEC))
def run():
for row in range(1, int(input()) + 1):
for m in MASTER.finditer(input()):
kind, value = m.lastgroup, m.group()
if kind in ("COMMENT", "SKIP"):
continue
where = f"{row}:{m.start() + 1}"
if kind == "MISMATCH":
print(where, "error")
return
if kind == "NAME" and value in KEYWORDS:
kind = "KEYWORD"
print(where, kind, value)
run()