Tokens and context windows

Intermediate

Introduction

A language model never sees letters or words directly. It sees tokens: pieces of text, each turned into a number. How text is cut into tokens decides how much a model can read at once, how much a request costs (services charge per token), and even which words it handles well. This lesson builds a small tokenizer and shows the limit every model works within, its context window.

How it works

A vocabulary is a fixed list of tokens, each with an ID. Encoding text means splitting it into tokens and replacing each with its ID; decoding is the reverse. The question is what a token should be:

  • Characters: a tiny vocabulary, but texts become very long sequences.
  • Whole words: short sequences, but the vocabulary must be enormous, and any new word (a name, a typo, "unhappily") is unknown.
  • Subwords: frequent words are single tokens, and rare words are built from pieces: "play" + "ed", "un" + "happi" + "ly". This is what modern models use, with vocabularies of roughly 30,000 to 200,000 pieces.

One simple way to split a word into subwords is greedy longest matching: from the start of the word, take the longest piece in the vocabulary, then continue after it. If no piece fits, emit an unknown token, written [UNK] here, for that character. (Byte-pair encoding, used by many models, builds its pieces by merging frequent pairs; real tokenizers also include every single byte, so nothing is ever unknown.)

A model's context window is the maximum number of tokens it can process at once: the instructions, the conversation so far, any documents, and its own answer must all fit. When a conversation grows too long, applications truncate: they drop the oldest messages (keeping the instructions), or summarize them. Whatever was dropped, the model simply cannot see.

As a rule of thumb, an English word is about 1.3 tokens on average, so 1000 tokens is around 750 words. Other languages and code often need more tokens per word.

Example

pieces = ["the", "cat", "s", "play", "ed", "un", "happi", "happy", "ness", "ly"]
vocab = {piece: i for i, piece in enumerate(["[UNK]"] + pieces)}


def tokenize_word(word):
    tokens, i = [], 0
    while i < len(word):
        for j in range(len(word), i, -1):  # try the longest piece first
            if word[i:j] in vocab:
                tokens.append(word[i:j])
                i = j
                break
        else:
            tokens.append("[UNK]")  # no piece fits this character
            i += 1
    return tokens


text = "the cats played unhappily"
tokens = [t for word in text.split() for t in tokenize_word(word)]
print("tokens:", tokens)
print("ids:   ", [vocab[t] for t in tokens])

window = 6
print(f"{len(tokens)} tokens; a window of {window} keeps the last {window}:", tokens[-window:])
Open in compiler

Line by line

  1. vocab gives each piece an ID; ID 0 is the unknown token.
  2. tokenize_word tries the longest slice first (range(len(word), i, -1)) and keeps the first that is in the vocabulary.
  3. The for ... else runs the else only when no piece matched, adding [UNK] and moving on by one character.
  4. "cats" becomes cat + s, "played" play + ed, and "unhappily" un + happi + ly; the sentence of 4 words becomes 8 tokens.
  5. A window of 6 tokens keeps only the last 6: the start of the sentence is gone.

Expected result

tokens: ['the', 'cat', 's', 'play', 'ed', 'un', 'happi', 'ly']
ids:    [1, 2, 3, 4, 5, 6, 7, 10]
8 tokens; a window of 6 keeps the last 6: ['s', 'play', 'ed', 'un', 'happi', 'ly']

In the real world

Every chat assistant counts tokens: prices are quoted per million tokens, and models are advertised by their context window, from a few thousand to hundreds of thousands or even millions of tokens. Applications that chat with long documents have to decide what to put in the window, which is the whole idea of retrieval (lesson 9). Tokenization also explains some odd behaviour, such as models miscounting the letters in a word: they see pieces, not letters.

Common mistakes

  • Counting words instead of tokens when estimating cost or context limits.
  • Assuming the model remembers everything said earlier in a long chat; truncated messages are gone.
  • Dropping the instructions when truncating, instead of the oldest conversation turns.
  • Mixing tokenizers: every model must be used with the tokenizer it was trained with.

Best practices

  • Measure token counts with the model's own tokenizer before sending long inputs.
  • Keep instructions at the start and trim old conversation turns first.
  • Leave room in the window for the model's answer.

Summary

Language models read tokens: pieces of text mapped to IDs by a fixed vocabulary. Subword tokens combine a manageable vocabulary with the ability to spell out any word, for example by greedy longest matching. The context window caps how many tokens a model can use at once, so long inputs must be truncated or selected, and anything left out is invisible to the model.

Quick check

1. Why do modern models use subword tokens?
2. Greedy longest matching with pieces 'play', 'p', 'ing' turns 'playing' into...
3. A chat no longer fits in the context window. What is usually dropped first?
4. About how many English words fit in 1000 tokens?

Key takeaways

  • Models read numbers: text becomes tokens, and tokens become IDs from a fixed vocabulary.
  • Subword tokens let a small vocabulary cover any word.
  • Everything a model uses must fit in its context window; the rest is invisible to it.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.