Fine-tuning and alignment

Advanced

Introduction

A language model that has only read the internet can continue any text, but it is not yet an assistant: asked a question, it may reply with another question, because that is what such text looks like. Turning a general model into a helpful and safe one takes more stages of training. This lesson explains them with ideas and tiny numbers: pretraining, fine-tuning, learning from human preferences, and LoRA, a trick that makes fine-tuning cheap.

How it works

Pretraining is the first and by far the most costly stage. The model predicts the next token over a huge amount of text and, in doing so, learns grammar, facts and many skills. It needs thousands of chips and months, so only a few organisations do it.

Fine-tuning continues training from those weights on a smaller, chosen set of examples: customer-support chats in a company's style, medical notes, or pairs of an instruction and a good reply (instruction tuning). The model keeps what it learned and shifts towards the new data. The example below shows this with counts: the fine-tuning text is added to the counts of the base model, so after "the" the word "order" becomes the most likely next word.

Preference training goes further. People are shown two answers to one prompt and mark the better one. Many such pairs make preference data. In RLHF (reinforcement learning from human feedback) a reward model learns to give the preferred answer the higher score; with a logistic formula the chance that it agrees with people is sigmoid(chosen score − rejected score), and the loss is minus the log of that chance. The language model is then trained to produce answers that the reward model scores highly. DPO (direct preference optimization) reaches a similar goal without a separate reward model, by training the language model directly on the preferred and rejected answers.

Fine-tuning every weight of a large model is expensive, because you must store and update billions of numbers. LoRA (low-rank adaptation) freezes the original weights and adds two thin matrices to a layer. A layer with d inputs and d outputs has d × d weights; LoRA of rank r trains only r × d + d × r = 2dr numbers. With d = 4096 and r = 8 that is 65,536 instead of 16,777,216, under 0.4 percent. The small add-on can be stored and swapped for each task.

Fine-tuning has risks too: too much training on a narrow set can make the model forget general skills, and it copies any mistakes or bias in the new data.

Example

import math

pretrain = "the cat sat . the dog ran . the cat ran . the sun is hot ."
finetune = "the order is late . the order is lost . the refund is sent ."


def counts(text, weight, table):
    words = text.split()
    for a, b in zip(words, words[1:]):
        table.setdefault(a, {})
        table[a][b] = table[a].get(b, 0) + weight
    return table


def top(table, word):
    after = table[word]
    total = sum(after.values())
    ranked = sorted(after.items(), key=lambda kv: (-kv[1], kv[0]))[:3]
    return ", ".join(f"{w} {c / total:.2f}" for w, c in ranked)


base = counts(pretrain, 1, {})
print("base model after 'the': ", top(base, "the"))
# Fine-tuning keeps what was learned and adds the new data, here counted twice
tuned = counts(finetune, 2, counts(pretrain, 1, {}))
print("tuned model after 'the':", top(tuned, "the"))

# Preference data: people preferred the chosen reply; a reward model scored both
chosen, rejected = 2.0, -1.0
p = 1 / (1 + round(math.exp(-(chosen - rejected)), 12))
print(f"reward model agrees with people with probability {p:.3f}, loss {round(-math.log(p), 12):.3f}")

# LoRA trains two thin matrices (d x r and r x d) instead of the full d x d matrix
d, r = 4096, 8
full, lora = d * d, r * (d + d)
print(f"one {d} x {d} layer: {full:,} weights; LoRA rank {r}: {lora:,} ({100 * lora / full:.2f}%)")
Open in compiler

Line by line

  1. Two small texts stand in for the huge pretraining set and the small fine-tuning set about orders and refunds.
  2. counts adds up how often each word follows another, multiplying by a weight.
  3. The base model, trained on the first text only, predicts cat, dog or sun after "the".
  4. The tuned model starts from the same counts and adds the new text twice. The word "order" is now the most likely, while the old knowledge is still there.
  5. The reward-model line applies the sigmoid to the gap between a chosen score of 2 and a rejected score of −1, giving a probability of about 0.953 that the model agrees with people.
  6. The last line counts parameters: full fine-tuning of one 4096 by 4096 layer against LoRA of rank 8.

Expected result

base model after 'the':  cat 0.50, dog 0.25, sun 0.25
tuned model after 'the': order 0.40, cat 0.20, refund 0.20
reward model agrees with people with probability 0.953, loss 0.049
one 4096 x 4096 layer: 16,777,216 weights; LoRA rank 8: 65,536 (0.39%)

In the real world

Chat assistants are built in exactly these stages: pretraining, instruction tuning, then preference training to make them helpful, honest and harmless. Companies fine-tune open models on their own documents or tone, often with LoRA so a single base model can serve many customers with small swappable add-ons. Fine-tuning does not replace retrieval: for facts that change, putting documents in the prompt (RAG) is simpler and cheaper than retraining.

Common mistakes

  • Fine-tuning to add facts that change often, where retrieval would work better.
  • Using too little or low-quality data, so the model copies its errors.
  • Training for too long, which makes the model forget general skills.
  • Testing on the training examples, which hides overfitting.
  • Treating human preferences as perfect, when they contain bias and disagreement.

Best practices

  • Try a better prompt, then retrieval, before you fine-tune.
  • Hold out test examples that the model never sees during fine-tuning.
  • Clean the data and check it for private details and bias.
  • Use LoRA or a similar method when memory and cost are limited.

Summary

Pretraining teaches a model general language from huge text; fine-tuning adapts it to a task or style with far less data. Preference data, a reward model (RLHF) or direct preference training (DPO) steer answers towards what people prefer. LoRA freezes the original weights and trains two thin matrices per layer, about 2dr numbers instead of d squared, which makes fine-tuning cheap.

Quick check

1. What is the main difference between pretraining and fine-tuning?
2. What does a reward model learn from preference data?
3. How many weights does LoRA of rank 2 train for a layer with 100 inputs and 50 outputs?
4. Your company's prices change every week. What is usually the better choice?

Key takeaways

  • Pretraining gives general ability; fine-tuning gives a specific skill, tone or format with far less data and compute.
  • Alignment uses human preferences, through a reward model (RLHF) or directly (DPO), to make answers more helpful and safe.
  • LoRA trains a small share of the weights, which makes fine-tuning cheap and easy to swap.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.