Introduction
A language model never sees letters or words directly. It sees tokens: pieces of text, each turned into a number. How text is cut into tokens decides how much a model can read at once, how much a request costs (services charge per token), and even which words it handles well. This lesson builds a small tokenizer and shows the limit every model works within, its context window.
How it works
A vocabulary is a fixed list of tokens, each with an ID. Encoding text means splitting it into tokens and replacing each with its ID; decoding is the reverse. The question is what a token should be:
- Characters: a tiny vocabulary, but texts become very long sequences.
- Whole words: short sequences, but the vocabulary must be enormous, and any new word (a name, a typo, "unhappily") is unknown.
- Subwords: frequent words are single tokens, and rare words are built from pieces: "play" + "ed", "un" + "happi" + "ly". This is what modern models use, with vocabularies of roughly 30,000 to 200,000 pieces.
One simple way to split a word into subwords is greedy longest matching: from the start of the word, take the longest piece in the vocabulary, then continue after it. If no piece fits, emit an unknown token, written [UNK] here, for that character. (Byte-pair encoding, used by many models, builds its pieces by merging frequent pairs; real tokenizers also include every single byte, so nothing is ever unknown.)
A model's context window is the maximum number of tokens it can process at once: the instructions, the conversation so far, any documents, and its own answer must all fit. When a conversation grows too long, applications truncate: they drop the oldest messages (keeping the instructions), or summarize them. Whatever was dropped, the model simply cannot see.
As a rule of thumb, an English word is about 1.3 tokens on average, so 1000 tokens is around 750 words. Other languages and code often need more tokens per word.
Example
pieces = ["the", "cat", "s", "play", "ed", "un", "happi", "happy", "ness", "ly"]
vocab = {piece: i for i, piece in enumerate(["[UNK]"] + pieces)}
def tokenize_word(word):
tokens, i = [], 0
while i < len(word):
for j in range(len(word), i, -1): # try the longest piece first
if word[i:j] in vocab:
tokens.append(word[i:j])
i = j
break
else:
tokens.append("[UNK]") # no piece fits this character
i += 1
return tokens
text = "the cats played unhappily"
tokens = [t for word in text.split() for t in tokenize_word(word)]
print("tokens:", tokens)
print("ids: ", [vocab[t] for t in tokens])
window = 6
print(f"{len(tokens)} tokens; a window of {window} keeps the last {window}:", tokens[-window:])Line by line
vocabgives each piece an ID; ID 0 is the unknown token.tokenize_wordtries the longest slice first (range(len(word), i, -1)) and keeps the first that is in the vocabulary.- The
for ... elseruns theelseonly when no piece matched, adding [UNK] and moving on by one character. - "cats" becomes cat + s, "played" play + ed, and "unhappily" un + happi + ly; the sentence of 4 words becomes 8 tokens.
- A window of 6 tokens keeps only the last 6: the start of the sentence is gone.
Expected result
tokens: ['the', 'cat', 's', 'play', 'ed', 'un', 'happi', 'ly']
ids: [1, 2, 3, 4, 5, 6, 7, 10]
8 tokens; a window of 6 keeps the last 6: ['s', 'play', 'ed', 'un', 'happi', 'ly']
In the real world
Every chat assistant counts tokens: prices are quoted per million tokens, and models are advertised by their context window, from a few thousand to hundreds of thousands or even millions of tokens. Applications that chat with long documents have to decide what to put in the window, which is the whole idea of retrieval (lesson 9). Tokenization also explains some odd behaviour, such as models miscounting the letters in a word: they see pieces, not letters.
Common mistakes
- Counting words instead of tokens when estimating cost or context limits.
- Assuming the model remembers everything said earlier in a long chat; truncated messages are gone.
- Dropping the instructions when truncating, instead of the oldest conversation turns.
- Mixing tokenizers: every model must be used with the tokenizer it was trained with.
Best practices
- Measure token counts with the model's own tokenizer before sending long inputs.
- Keep instructions at the start and trim old conversation turns first.
- Leave room in the window for the model's answer.
Summary
Language models read tokens: pieces of text mapped to IDs by a fixed vocabulary. Subword tokens combine a manageable vocabulary with the ability to spell out any word, for example by greedy longest matching. The context window caps how many tokens a model can use at once, so long inputs must be truncated or selected, and anything left out is invisible to the model.
Quick check
Key takeaways
- Models read numbers: text becomes tokens, and tokens become IDs from a fixed vocabulary.
- Subword tokens let a small vocabulary cover any word.
- Everything a model uses must fit in its context window; the rest is invisible to it.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.