Compilers: Tokens with line and column

Lex a multi-line program and record where every token starts.

What to build

A program that tokenizes several lines and prints every token with its position.

Requirements

  • Rules: STRING ("…" with no quote inside), NUMBER, NAME/KEYWORD (if, else, while, print, def, return), OP (==, !=, <=, >=, then one of + - * / = < > ( ) , :), # comments, whitespace.
  • Print line:column KIND lexeme. On an unexpected character print line:column error and stop the whole program.

Input

Line 1: n. Then n lines of code.

Output

One line per token, possibly ending with an error line.

Constraints

n ≤ 10; lines have at most 80 characters.

Example

Input:

2
def area(r):
    return 3.14 * r * r

Output:

1:1 KEYWORD def
1:5 NAME area
1:9 OP (
1:10 NAME r
1:11 OP )
1:12 OP :
2:5 KEYWORD return
2:12 NUMBER 3.14
2:17 OP *
2:19 NAME r
2:21 OP *
2:23 NAME r

Why it works

Lexing line by line and keeping each match's start position gives exact locations, which every later error message depends on.

Examples

Example 1 · Input
2
def area(r):
    return 3.14 * r * r
Expected output
1:1 KEYWORD def
1:5 NAME area
1:9 OP (
1:10 NAME r
1:11 OP )
1:12 OP :
2:5 KEYWORD return
2:12 NUMBER 3.14
2:17 OP *
2:19 NAME r
2:21 OP *
2:23 NAME r
Example 2 · Input
1
print("hi", x)
Expected output
1:1 KEYWORD print
1:6 OP (
1:7 STRING "hi"
1:11 OP ,
1:13 NAME x
1:14 OP )

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Run the master pattern on each line separately, counting lines from 1, and use m.start() + 1 for the column.

Hint 2

Put the STRING rule (double quotes, no newline inside) before the operators.

Solution

Show one correct solution (try it yourself first)
import re

KEYWORDS = {"if", "else", "while", "print", "def", "return"}
SPEC = [
    ("STRING", r'"[^"\n]*"'),
    ("NUMBER", r"\d+(\.\d+)?"),
    ("NAME", r"[A-Za-z_]\w*"),
    ("OP", r"==|!=|<=|>=|[-+*/=<>(),:]"),
    ("COMMENT", r"#.*"),
    ("SKIP", r"\s+"),
    ("MISMATCH", r"."),
]
MASTER = re.compile("|".join(f"(?P<{n}>{p})" for n, p in SPEC))


def run():
    for row in range(1, int(input()) + 1):
        for m in MASTER.finditer(input()):
            kind, value = m.lastgroup, m.group()
            if kind in ("COMMENT", "SKIP"):
                continue
            where = f"{row}:{m.start() + 1}"
            if kind == "MISMATCH":
                print(where, "error")
                return
            if kind == "NAME" and value in KEYWORDS:
                kind = "KEYWORD"
            print(where, kind, value)


run()

Lesson: Lexical analysis with regular expressions →