Transformers

Advanced

Introduction

The transformer, introduced in 2017 in the paper "Attention Is All You Need", replaced recurrent networks for language and has since spread to images, audio, proteins and more. GPT-style chatbots, translation systems and code assistants are all transformers. This lesson puts together the pieces from the earlier lessons, embeddings, attention, layer normalization, residual connections and softmax, into the complete architecture.

How it works

Input. Text is split into tokens, and each token is looked up in an embedding table. Attention by itself treats its inputs as a set, with no order, so a positional encoding is added to each embedding. The original transformer used fixed sine and cosine waves of different frequencies: for position p and dimension pair i, PE[p][2i] = sin(p / 10000^(2i/d)) and PE[p][2i + 1] = cos(p / 10000^(2i/d)). Many models learn position embeddings instead.

Blocks. The model is a stack of identical transformer blocks. Each block has two sub-layers:

  1. multi-head self-attention, which lets each position gather information from the others (with a causal mask for text generation);
  2. a feed-forward network, two dense layers with a non-linearity, usually 4 times wider than d, applied to each position on its own.

Each sub-layer is wrapped in a residual connection (output = input + sublayer(input)) and layer normalization. Residual connections let gradients flow straight through dozens of blocks, which is what makes very deep transformers trainable.

Output. After the last block, a final layer norm and a linear layer produce a logit for every token in the vocabulary, often reusing (tying) the embedding table's weights. Softmax gives the probability of each possible next token, and training minimizes the cross-entropy of the actual next token at every position.

Generating text. Start with a prompt, compute the next-token probabilities, choose a token, append it and repeat. Greedy decoding always takes the most likely token. Beam search keeps the B most likely partial sequences at each step, so it can find a sequence whose first token was not the single best choice but whose total probability is higher. Chat models usually sample instead, with a temperature that controls how adventurous the choices are.

Syntax

x = [embed[t] + positional(i) for i, t in enumerate(tokens)]
for block in blocks:
    x = layer_norm(x + self_attention(x))      # residual + norm
    x = layer_norm(x + feed_forward(x))
probabilities = softmax(x[-1] @ embed.T)        # next-token prediction

Example

import math

d = 4
tokens = ["I", "like", "tea"]
embedding = {"I": [1.0, 0.0, 0.0, 1.0], "like": [0.0, 1.0, 1.0, 0.0], "tea": [1.0, 1.0, 0.0, 0.0]}


def positional(pos):
    return [math.sin(pos / 10000 ** (i / d)) if i % 2 == 0 else math.cos(pos / 10000 ** ((i - 1) / d)) for i in range(d)]


def layer_norm(v):
    mean = sum(v) / len(v)
    var = sum((x - mean) ** 2 for x in v) / len(v)
    return [(x - mean) / math.sqrt(var + 1e-5) for x in v]


def softmax(z):
    top = max(z)
    e = [round(math.exp(v - top), 12) for v in z]  # rounded, so every computer gives the same digits
    return [v / sum(e) for v in e]


x = [[e + p for e, p in zip(embedding[t], positional(i))] for i, t in enumerate(tokens)]
attended = []
for i in range(len(x)):
    w = softmax([sum(a * b for a, b in zip(x[i], x[j])) / math.sqrt(d) for j in range(i + 1)])
    attended.append([sum(w[j] * x[j][c] for j in range(i + 1)) for c in range(d)])
x = [layer_norm([a + b for a, b in zip(row, att)]) for row, att in zip(x, attended)]
ffn = [layer_norm([a + max(0.0, 0.5 * a) for a in row]) for row in x]
print("block output for each token:")
for t, row in zip(tokens, ffn):
    print(f"    {t:5}", " ".join(f"{v:6.3f}" for v in row))

next_token = {"I": {"like": 0.6, "drink": 0.4}, "like": {"tea": 0.5, "coffee": 0.3, "END": 0.2}, "drink": {"tea": 0.9, "END": 0.1}, "tea": {"END": 1.0}, "coffee": {"END": 1.0}}
sequence, word, probability = ["I"], "I", 1.0
while word != "END":
    word, p = max(next_token[word].items(), key=lambda item: item[1])
    sequence.append(word)
    probability *= p
print("greedy generation:", " ".join(sequence), f"(probability {probability:.2f})")
Open in compiler

Line by line

  1. Three tokens are embedded in 4 dimensions and given sine and cosine positional encodings.
  2. Causal self-attention mixes each position with itself and earlier positions.
  3. Residual connections add the attention output to the input, and layer normalization rescales each position.
  4. A tiny feed-forward step (with ReLU) and another residual and normalization complete the block.
  5. A toy next-token table shows greedy generation: from "I" it picks "like", then "tea", then END.

Expected result

block output for each token:
    I     -0.140 -0.140 -1.260  1.540
    like  -0.816  1.446 -1.038  0.407
    tea    1.382 -0.414 -1.336  0.368
greedy generation: I like tea END (probability 0.30)

In the real world

GPT-2 small, with 12 blocks of width 768, has about 124 million parameters; today's large models have tens or hundreds of billions, but the block is recognisably the same. Changes since 2017 include placing layer norm before each sub-layer, rotary position embeddings, and mixture-of-experts feed-forward layers that activate only part of the network for each token. The training objective, predicting the next token with cross-entropy, has stayed the same.

Common mistakes

  • Forgetting positional information, which makes "dog bites man" and "man bites dog" identical to the model.
  • Leaving out the residual connections, which makes deep stacks hard to train.
  • Using greedy decoding for everything; it can produce repetitive or poor sequences.
  • Miscounting parameters, for example forgetting biases or counting tied embeddings twice.

Best practices

  • Count parameters per component to see where the model's size really goes.
  • Check generation with a tiny, fully known next-token table before using a real model.
  • Use beam search when one best output is wanted (translation) and sampling for open-ended text.

Summary

A transformer adds positional encodings to token embeddings and passes them through a stack of blocks, each with multi-head self-attention and a feed-forward network, wrapped in residual connections and layer normalization. A final softmax gives next-token probabilities, and decoding methods such as greedy search and beam search turn them into text.

Quick check

1. Why do transformers need positional encodings?
2. What does a residual connection compute?
3. What does the feed-forward network in a block do?
4. How can beam search beat greedy decoding?

Key takeaways

  • Attention ignores order, so positions must be added to the embeddings.
  • A transformer is a stack of identical blocks: attention mixes information between positions, the feed-forward network processes each position.
  • Text is generated one token at a time from the probabilities of the next token.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.