Logistic regression

Despite its name, logistic regression is a classifier: it predicts the probability that an example belongs to the positive class (pass, spam, fraud).

The sigmoid

Start like linear regression, with a score z = w × x + b. A score can be any number, but a probability must lie between 0 and 1, so squash z with the sigmoid (logistic) function:

σ(z) = 1 / (1 + e^(−z))

Big positive scores give probabilities near 1, big negative ones near 0, and z = 0 gives exactly 0.5. Predict 1 when σ(z) ≥ 0.5, that is, when z ≥ 0. The place where z = 0 is the decision boundary: with one feature it is a single point x = −b/w; with two it is a line; with more, a flat plane.

The log loss

For classification we use the log loss (cross-entropy) instead of the squared error. For one example with label y (0 or 1) and predicted probability p:

loss = −( y × log p + (1 − y) × log(1 − p) )

When y = 1, the loss is −log p: tiny if p is near 1, huge if the model was confidently wrong (p near 0). That strong push is what makes the model learn.

Training

Gradient descent again, and the gradient turns out as neat as linear regression's:

∂L/∂w = mean of (pᵢ − yᵢ) × xᵢ
∂L/∂b = mean of (pᵢ − yᵢ)

from math import exp, log

hours  = [1, 2, 3, 4, 5, 6]
passed = [0, 0, 0, 1, 1, 1]
n = len(hours)
mean = sum(hours) / n          # 3.5: the feature is centred, so training settles fast

def sigmoid(z):
    return 1 / (1 + exp(-z))

w, b, rate = 0.0, 0.0, 0.5
for step in range(20):
    dw = db = 0.0
    for h, y in zip(hours, passed):
        p = sigmoid(w * (h - mean) + b)
        dw += (p - y) * (h - mean) / n
        db += (p - y) / n
    w, b = w - rate * dw, b - rate * db

def chance(h):
    return sigmoid(w * (h - mean) + b)

loss = -sum(y * log(chance(h)) + (1 - y) * log(1 - chance(h)) for h, y in zip(hours, passed)) / n
print(f"w = {w:.3f}, log loss = {loss:.3f}, boundary at {mean - b / w:.2f} hours")
for h in [2, 3.5, 5]:
    print(h, "hours ->", f"{chance(h):.0%} chance to pass")

The program first centres the feature (hours minus their mean, 3.5), as the lesson on data advised: gradient descent then settles in a few steps instead of swinging back and forth. After training, the model gives a probability for any number of hours, not just a yes or no. The boundary, where the probability crosses 0.5, falls at 3.5 hours, right between the last student who failed and the first who passed.

More classes

For more than two classes, softmax regression gives each class its own score and turns the scores into probabilities that add up to 1. It is the last layer of almost every neural network classifier.

Mark lesson complete