Attention, transformers and LLMs in practice

Advanced

Introduction

Every lesson so far looked at a few neighbouring words at a time: n-grams, a window of co-occurrence, an HMM's previous tag. Language needs more: in "The animal didn't cross the street because it was too tired", the word "it" depends on "animal", eight words back. Attention lets every token look directly at every other token and decide which ones matter. The transformer, built from attention, is the architecture behind BERT, GPT, Llama, Claude and every other large language model (LLM). This lesson computes attention by hand, shows how an LLM chooses its next word, and ends with how these models are used in practice.

How it works

Self-attention. Each token has a vector. From it the model computes three vectors with learned weight matrices: a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?). For token i:

  1. Score every token j: sᵢⱼ = qᵢ · kⱼ ÷ √d, where d is the vector size. Dividing by √d stops scores from growing with d and saturating the softmax.
  2. Turn the scores into weights with softmax: wᵢⱼ = e^(sᵢⱼ) ÷ Σ e^(sᵢₖ). The weights are positive and add up to 1. (Subtract the largest score first; the result is the same and no exponential overflows.)
  3. The output is the weighted average of the value vectors: Σ wᵢⱼ vⱼ.

The example uses each token's vector itself as query, key and value, the simplest form. A causal mask lets token i attend only to tokens 0 to i; generating models need it, because the next word must not be visible while it is being predicted. Multi-head attention runs several attentions side by side, each free to track a different relation (grammar, coreference, nearby words).

Transformers. A transformer block applies self-attention, then a small feed-forward network to each token, with residual connections and normalization around both. Positional encodings are added to the input vectors, because attention by itself ignores word order. An LLM stacks dozens of such blocks on top of a token embedding table, and its final layer gives a score, a logit, for every token in the vocabulary.

Decoding. Softmax turns the logits into next-token probabilities. Temperature T divides the logits first: T below 1 sharpens the distribution (safer, more repetitive text), T above 1 flattens it (more varied, more mistakes). Then a token is chosen:

  • Greedy: always the most likely token.
  • Top-k: keep the k most likely tokens, renormalize their probabilities, and sample.
  • Top-p (nucleus): keep the smallest set of most likely tokens whose probabilities add up to at least p, renormalize, and sample.

The chosen token is appended and the model runs again, one token at a time.

Example

import math

tokens = ["the", "cat", "sat"]
x = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
d = len(x[0])


def softmax(scores):
    top = max(scores)
    e = [round(math.exp(s - top), 12) for s in scores]
    total = sum(e)
    return [v / total for v in e]


for i, q in enumerate(x):
    scores = [round(sum(a * b for a, b in zip(q, k)) / math.sqrt(d), 12) for k in x]
    weights = softmax(scores)
    out = [sum(w * v[j] for w, v in zip(weights, x)) for j in range(d)]
    print(f"{tokens[i]}: weights {[round(w, 3) for w in weights]} -> {[round(o, 3) for o in out]}")

logits = {"mat": 3.0, "floor": 2.0, "sofa": 1.5, "moon": 0.0}
print("greedy choice:", max(logits, key=logits.get))
for temperature in (1.0, 0.5, 2.0):
    probs = softmax([z / temperature for z in logits.values()])
    print(f"T={temperature}:", "  ".join(f"{w} {p:.3f}" for w, p in zip(logits, probs)))
Open in compiler

Line by line

  1. The three tokens have 2-number vectors, so d = 2 and every dot product is divided by √2.
  2. "the" matches itself and "sat" equally (both dot products are 1) and "cat" not at all, so it gives 0.401 to each of them and only 0.198 to "cat".
  3. "sat" matches itself best (dot product 2), so half of its attention, 0.503, stays on itself.
  4. Each output is a weighted average of the vectors, so every token's new vector now mixes in information from the others.
  5. The logits give "mat" the highest probability. Temperature 0.5 makes it dominate (0.842); temperature 2.0 spreads the probability out, and even "moon" gets almost 10%.

Expected result

the: weights [0.401, 0.198, 0.401] -> [0.802, 0.599]
cat: weights [0.198, 0.401, 0.401] -> [0.599, 0.802]
sat: weights [0.248, 0.248, 0.503] -> [0.752, 0.752]
greedy choice: mat
T=1.0: mat 0.609  floor 0.224  sofa 0.136  moon 0.030
T=0.5: mat 0.842  floor 0.114  sofa 0.042  moon 0.002
T=2.0: mat 0.434  floor 0.263  sofa 0.205  moon 0.097

In the real world

Using a pretrained model takes a few lines with Hugging Face's transformers (shown, not run: it needs the library and downloads a model, which StepVioz's browser Python cannot do):

from transformers import pipeline

generator = pipeline("text-generation", model="gpt2")
print(generator("The cat sat on the", max_new_tokens=5, do_sample=True, temperature=0.7, top_k=50))

In practice, most applications call a large model through an API or run an open model locally, and the craft moves to the input and the checks around it:

  • Prompting: clear instructions, examples of the wanted output (few-shot), and a fixed output format.
  • Retrieval-augmented generation (RAG): search your own documents with embeddings or TF-IDF, put the best passages into the prompt, and ask the model to answer from them. This is the search lesson and the embeddings lesson working together.
  • Fine-tuning: further training on your own examples when prompting is not enough.
  • Evaluation: test sets with expected answers, scored with the precision, recall and accuracy ideas of this course, plus human review.

LLMs can state false things fluently (hallucination), repeat biases from their training text, and leak data placed in prompts, so production systems check outputs, cite sources and keep people involved in important decisions. The Deep Learning course builds attention, transformer blocks and training from scratch, if you want to go deeper.

Common mistakes

  • Forgetting the √d scaling, which makes softmax nearly one-hot for large vectors and slows learning.
  • Computing softmax without subtracting the maximum, which overflows for large logits.
  • Leaving out the causal mask when training a model to generate text, so it learns to copy the answer from the future.
  • Treating temperature 0 or greedy decoding as 'more correct': it is only more predictable, and it can loop on repeated phrases.

Best practices

  • Check that every row of attention weights adds up to 1, and that masked weights are exactly 0.
  • Use low temperature for factual or structured output and higher temperature for brainstorming.
  • Ground answers in retrieved documents and ask the model to quote them when accuracy matters.
  • Evaluate an LLM feature on a fixed set of real examples before and after every change of prompt or model.

Summary

Self-attention lets each token build a new vector as a weighted average of all tokens, with weights from a softmax over scaled query-key dot products; a causal mask hides later tokens. Transformers stack attention and feed-forward layers, and an LLM predicts the next token from logits. Temperature, greedy, top-k and top-p decoding turn those logits into text, and in practice LLMs are used with prompting, retrieval, fine-tuning and careful evaluation.

Quick check

1. Why are attention scores divided by √d?
2. What does a causal mask prevent?
3. What happens to next-token probabilities at temperature 0.5 compared with 1?
4. What does top-p sampling with p = 0.9 keep?

Key takeaways

  • Attention mixes information between tokens with weights that depend on the tokens themselves.
  • A large language model is a stack of transformer blocks trained to predict the next token.
  • Decoding settings such as temperature, top-k and top-p change the text a model produces without changing the model.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.