Softmax and multi-class classification

Intermediate

Introduction

Many tasks have more than two answers: which digit is in an image, which of 50,000 words comes next, which language a sentence is in. The network outputs one score per class, called logits, and softmax turns them into probabilities. Trained with cross-entropy, this combination is the standard output layer of image classifiers and language models alike.

How it works

Softmax exponentiates each logit and divides by the total:

pᵢ = e^(zᵢ) / Σⱼ e^(zⱼ)

The exponentials are positive, so every pᵢ is positive, and dividing by the total makes them add up to 1. A larger logit always gets a larger probability, and the gaps grow exponentially: logits 1, 2, 3 give about 0.09, 0.24 and 0.67.

Numerical stability: e^1000 overflows. But adding the same number to every logit does not change softmax, because it multiplies the top and bottom by the same factor. So real code subtracts the largest logit first; the biggest exponent is then e^0 = 1.

Cross-entropy for one example is −log p_y, where y is the true class: near 0 when the model is confident and right, very large when it is confident and wrong. Averaged over the data, it is the loss to minimise. A model that knows nothing and gives every class 1/K has loss log K.

The gradient is remarkably simple. For the logits z:

∂L/∂zᵢ = pᵢ − 1 if i is the true class, else pᵢ

that is, p minus the one-hot vector. For a linear classifier z = W x + b, the chain rule gives ∂L/∂W = (p − one-hot) xᵀ and ∂L/∂b = p − one-hot, averaged over the batch. This is softmax regression (multinomial logistic regression); putting hidden layers before it makes a deep classifier.

Syntax

def softmax(z):
    top = max(z)
    e = [math.exp(v - top) for v in z]
    return [v / sum(e) for v in e]

loss = -math.log(p[label])
grad = [p_k - (1 if k == label else 0) for k, p_k in enumerate(p)]

Example

import math


def softmax(logits):
    top = max(logits)
    exps = [round(math.exp(z - top), 12) for z in logits]  # rounded, so every computer gives the same digits
    total = sum(exps)
    return [e / total for e in exps]


print("softmax([1, 2, 3]) =", [round(p, 4) for p in softmax([1, 2, 3])])
print("softmax([1000, 1001]) =", [round(p, 4) for p in softmax([1000, 1001])])

points = [([2.0, 0.5], 0), ([1.5, 1.0], 0), ([-1.0, 2.0], 1), ([-1.5, 1.0], 1), ([0.0, -2.0], 2), ([0.5, -1.5], 2)]
K, D, RATE = 3, 2, 0.5
W = [[0.0] * D for _ in range(K)]
b = [0.0] * K
for epoch in range(1, 101):
    loss = 0.0
    dW = [[0.0] * D for _ in range(K)]
    db = [0.0] * K
    for x, label in points:
        p = softmax([sum(w * xi for w, xi in zip(W[k], x)) + b[k] for k in range(K)])
        loss -= round(math.log(p[label]), 12) / len(points)
        for k in range(K):
            g = (p[k] - (1 if k == label else 0)) / len(points)
            db[k] += g
            for j in range(D):
                dW[k][j] += g * x[j]
    for k in range(K):
        b[k] -= RATE * db[k]
        for j in range(D):
            W[k][j] -= RATE * dW[k][j]
    if epoch in (1, 10, 100):
        print(f"epoch {epoch:3}: loss {loss:.4f}")

test = [1.0, 1.0]
probs = softmax([sum(w * xi for w, xi in zip(W[k], test)) + b[k] for k in range(K)])
print("point (1, 1): class probabilities", [round(p, 3) for p in probs], "-> class", probs.index(max(probs)))
Open in compiler

Line by line

  1. softmax subtracts the largest logit, so even logits of 1000 and 1001 work.
  2. Six points belong to three classes: right, upper left and below.
  3. Starting from zero weights, every class has probability 1/3, so the first loss is log 3 ≈ 1.0986.
  4. Each epoch averages the gradient p − one-hot times the input over all points and steps against it.
  5. After 100 epochs the loss is small, and a new point at (1, 1) is assigned most probability for class 0, whose points lie to the right.

Expected result

softmax([1, 2, 3]) = [0.09, 0.2447, 0.6652]
softmax([1000, 1001]) = [0.2689, 0.7311]
epoch   1: loss 1.0986
epoch  10: loss 0.1100
epoch 100: loss 0.0110
point (1, 1): class probabilities [0.937, 0.053, 0.01] -> class 0

In the real world

Image classifiers end with a softmax over 1,000 or more classes; language models end with a softmax over their whole vocabulary to predict the next token, and their training loss is exactly this cross-entropy. Frameworks provide a fused "softmax with cross-entropy" function, which is both faster and more stable than doing the two steps separately.

Common mistakes

  • Exponentiating large logits directly, which overflows to infinity and gives NaN.
  • Applying softmax twice, which flattens the probabilities.
  • Taking log(0) when a probability underflows; working with log-probabilities avoids it.
  • Using the class index as a number to regress on, instead of one probability per class.

Best practices

  • Always subtract the maximum logit before exponentiating.
  • Check that the first loss of an untrained model is about log K.
  • Report accuracy as well as loss; they tell different stories.

Summary

Softmax turns logits into probabilities by exponentiating and normalizing, and subtracting the largest logit keeps it stable. Cross-entropy is −log of the true class's probability. Their combined gradient is p − one-hot, which makes training a softmax classifier straightforward.

Quick check

1. What do softmax outputs always add up to?
2. Why subtract the largest logit before exponentiating?
3. What is the gradient of cross-entropy with respect to the logits?
4. What loss should an untrained model with 10 equally likely classes have?

Key takeaways

  • Softmax keeps the order of the logits and makes them a probability distribution.
  • Subtracting the largest logit before exponentiating gives the same result without overflow.
  • With cross-entropy, the gradient with respect to the logits is simply p − one-hot.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.