Introduction
A language model gives a probability for every possible next token, but it does not say which one to use. That choice, decoding, has a big effect on the result: the same model can sound robotic, creative or confused depending on its settings. Chat applications expose some of these settings, such as temperature, and this lesson shows exactly what they do.
How it works
Greedy decoding always takes the most likely token. It is repeatable, but it often produces bland, repetitive text, and it can miss a sequence whose first token was slightly less likely but whose whole is much better.
Sampling chooses at random with the model's probabilities, which gives variety. Several settings control it:
- Temperature T divides every logit before softmax: P ∝ e^(z / T). With T below 1 the gaps between scores grow, so the top tokens dominate; at T close to 0 it becomes greedy. With T above 1 the distribution flattens, so unlikely tokens get more chances. Low temperatures suit facts and code; higher ones suit brainstorming and stories.
- Top-k keeps only the k most likely tokens, renormalizes their probabilities to add up to 1, and samples among them.
- Top-p (nucleus) keeps the smallest set of most likely tokens whose probabilities add up to at least p, then renormalizes. When the model is confident, the set is small; when it is unsure, it grows.
Beam search keeps the B most probable partial sequences (the beam) instead of one. At each step every sequence is extended with every possible next token, the probability of each new sequence is the product of its token probabilities, and only the best B survive. It finds more probable sequences than greedy decoding and suits tasks with one right answer, such as translation; for open-ended writing, sampling usually reads better.
Example
import math
words = ["blue", "grey", "clear", "green", "falling"]
logits = [3.0, 2.0, 1.5, 0.5, -1.0] # a model's scores for the word after "The sky is"
def softmax(scores, temperature=1.0):
scaled = [s / temperature for s in scores]
top = max(scaled)
exps = [round(math.exp(s - top), 12) for s in scaled] # rounded, so every computer agrees
total = sum(exps)
return [e / total for e in exps]
for t in [0.5, 1.0, 2.0]:
print(f"temperature {t}:", [round(p, 2) for p in softmax(logits, t)])
ranked = sorted(zip(words, softmax(logits)), key=lambda wp: -wp[1])
print("greedy:", ranked[0][0])
print("top-k (k = 2):", [w for w, _ in ranked[:2]])
kept, running = [], 0.0
for w, p in ranked:
kept.append(w)
running += p
if running >= 0.9:
break
print("top-p (p = 0.9):", kept)Line by line
- The logits are a model's scores for the word after "The sky is".
softmaxdivides the logits by the temperature before exponentiating.- At temperature 0.5 almost all the probability goes to "blue"; at 2.0 the other words get far more.
- Greedy decoding picks the top word; top-k with k = 2 keeps the two best.
- Top-p adds words from the most likely down until their total reaches 0.9.
Expected result
temperature 0.5: [0.84, 0.11, 0.04, 0.01, 0.0]
temperature 1.0: [0.59, 0.22, 0.13, 0.05, 0.01]
temperature 2.0: [0.4, 0.24, 0.19, 0.11, 0.05]
greedy: blue
top-k (k = 2): ['blue', 'grey']
top-p (p = 0.9): ['blue', 'grey', 'clear']
In the real world
Chat assistants typically sample with a moderate temperature and top-p, so answers vary a little each time. Code assistants and data-extraction tools use low temperatures for predictable output. Machine translation and speech recognition have long used beam search. Applications also add penalties for repeating recent tokens, which break the loops that greedy decoding falls into.
Common mistakes
- Using a high temperature for factual tasks, which raises the chance of invented details.
- Expecting temperature 0 to make a model correct. It makes it consistent, not right.
- Forgetting to renormalize after top-k or top-p, so the kept probabilities no longer add up to 1.
- Using beam search for creative writing, where it tends to produce generic, repetitive text.
Best practices
- Choose low temperatures for extraction, classification and code; higher ones for ideas and stories.
- Combine temperature with top-p rather than relying on temperature alone.
- Fix the random seed when comparing settings, so differences come from the settings.
Summary
Decoding turns next-token probabilities into text. Greedy decoding takes the top token; sampling chooses randomly by probability, shaped by temperature (sharper below 1, flatter above 1) and restricted by top-k or top-p to the likely tokens. Beam search keeps the B most probable partial sequences and suits tasks with one correct output.
Quick check
Key takeaways
- Greedy decoding is repeatable but dull and loop-prone; sampling adds variety.
- Low temperature makes output focused and predictable; high temperature makes it diverse and riskier.
- Top-k and top-p cut off the unlikely tail; beam search looks for the most probable whole sequence.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.