Perceptrons and neural networks

One neuron

An artificial neuron takes inputs x₁, ..., xₙ, multiplies each by a weight, adds a bias, and passes the sum through an activation function:

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

The perceptron (Rosenblatt, 1958) uses a step: output 1 if the sum is above 0, else 0. Logistic regression is the same neuron with a sigmoid instead of a step.

The perceptron learning rule

For each training example, compare the output ŷ with the label y, and nudge the weights:

wᵢ ← wᵢ + α × (y − ŷ) × xᵢ
b  ← b  + α × (y − ŷ)

When the output is right, y − ŷ = 0 and nothing changes. When it said 0 but should have said 1, the weights of the active inputs grow, and vice versa.

examples = [((0, 0), 0), ((0, 1), 0), ((1, 0), 0), ((1, 1), 1)]   # AND
w = [0, 0]
b = 0
rate = 1

def predict(x):
    return 1 if w[0] * x[0] + w[1] * x[1] + b > 0 else 0

for epoch in range(1, 11):
    mistakes = 0
    for x, y in examples:
        error = y - predict(x)
        if error != 0:
            mistakes += 1
            w = [w[0] + rate * error * x[0], w[1] + rate * error * x[1]]
            b += rate * error
    print(f"epoch {epoch}: w = {w}, b = {b}, mistakes {mistakes}")
    if mistakes == 0:
        break

The perceptron learns the AND function in a few passes (epochs) over the four examples. If the classes can be split by a straight line (they are linearly separable), the rule is guaranteed to find such a line.

XOR: the limit of one neuron

XOR is 1 when exactly one input is 1. Plot its four points, and no straight line separates the 1s from the 0s, so no single perceptron can learn it. This was pointed out in 1969 and slowed neural-network research for over a decade.

def step(z):
    return 1 if z > 0 else 0

def neuron(weights, bias, inputs):
    return step(sum(w * x for w, x in zip(weights, inputs)) + bias)

# Two hidden neurons and one output neuron, with weights chosen by hand
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h_or = neuron([1, 1], -0.5, x)          # OR
    h_nand = neuron([-1, -1], 1.5, x)       # NAND
    out = neuron([1, 1], -1.5, [h_or, h_nand])   # AND of the two
    print(x, "-> hidden", (h_or, h_nand), "-> XOR", out)

The fix is a hidden layer. XOR is "OR, but not AND": one hidden neuron computes OR, another computes NAND, and an output neuron ANDs them. Layers of neurons can build any function out of simple pieces.

Neural networks

A multilayer perceptron has an input layer, one or more hidden layers and an output layer, with every neuron connected to every neuron of the next layer.

• Activation functions must be non-linear, or the whole network collapses into one linear function. Common ones: sigmoid, tanh, and ReLU, max(0, z), the usual choice today.
• Backpropagation trains the network by gradient descent. The chain rule of calculus carries each output error back through the layers, telling every weight how much it was to blame.
• Deep learning means networks with many layers, trained on huge datasets with GPUs. Special designs suit special data: convolutional networks for images, and transformers for text and speech; the large language models behind chat assistants are transformers.

The Deep Learning course, in Phase 5, builds these.

Mark lesson complete