From source to tokens
What a compiler is made of, and its first phase: turning characters into tokens, with regular expressions and with a hand-written scanner.
About 4 hours of study
Learning objectives
- Name the phases of a compiler and what each produces
- Write a lexer from a list of regular expressions
- Write a character-by-character lexer with positions and errors
Lessons
- 1
How a compiler works Intermediate
A compiler turns source code into something a machine can run, in phases: lexing, parsing, checking, intermediate code, optimisation and code generation. Build a whole tiny compiler for arithmetic in 60 lines of Python.
- 2
Lexical analysis with regular expressions Intermediate
A lexer turns characters into tokens: numbers, names, keywords and operators, with their positions. Build one from a table of regular expressions combined into a single pattern, the way many real lexers are generated.
- 3
A hand-written lexer Intermediate
Write a lexer the way production compilers often do: one character at a time, with peek and advance, longest match, and lookahead for two-character operators and decimal numbers.
Module practice
- Compilers: Split an expression into tokensEasy
- Compilers: Run postfix code on a stackMedium
- Compilers: A whole tiny compilerHard
- Compilers: Classify lexemesEasy
- Compilers: A lexer from a rule listMedium
- Compilers: Tokens with line and columnHard
- Compilers: Pick out the numbersEasy
- Compilers: Longest-match operatorsMedium
- Compilers: A complete hand-written lexerHard