Introduction
One neuron draws one straight boundary, so it can learn AND or OR but never XOR (true when exactly one input is true): no straight line separates {01, 10} from {00, 11}. This limitation stalled neural-network research in the 1970s. The fix is a hidden layer: neurons whose outputs are new features that the output neuron combines. This lesson trains such a network from scratch.
How it works
A multi-layer perceptron (MLP) stacks layers: inputs → hidden layer(s) → output. Each hidden neuron learns a feature, such as "at least one input is on" or "both inputs are on", and the output neuron combines the features: XOR is "at least one, but not both".
The training loop repeats, for each batch of examples:
- forward pass: compute every layer's activations and the prediction;
- loss: compare the prediction with the target;
- backward pass: backpropagate to get every gradient;
- update: move each parameter against its gradient.
One pass over the whole dataset is an epoch. Updating after every example is stochastic gradient descent; averaging gradients over the whole dataset is full-batch gradient descent; most real training uses mini-batches of 32–1,024 examples.
The number of parameters grows quickly: a layer with n inputs and m neurons has n × m weights and m biases. A 784-128-10 network for 28 × 28 images already has 101,770 parameters.
Training is not guaranteed to succeed. From an unlucky start, or with too few hidden neurons, it can get stuck with some examples wrong. In practice, slightly larger networks, sensible initialization and better optimizers make this rare.
Syntax
for epoch in range(epochs):
for x, y in data:
h, p = forward(x) # 1. forward
loss = (p - y) ** 2 # 2. loss
grads = backward(x, h, p, y) # 3. backward
update(grads, rate) # 4. update
Example
import math
import random
random.seed(3)
DATA = [([0.0, 0.0], 0.0), ([0.0, 1.0], 1.0), ([1.0, 0.0], 1.0), ([1.0, 1.0], 0.0)]
HIDDEN, RATE = 3, 0.5
W1 = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(HIDDEN)]
b1 = [0.0] * HIDDEN
W2 = [random.uniform(-1, 1) for _ in range(HIDDEN)]
b2 = 0.0
def sigmoid(z):
return 1 / (1 + round(math.exp(-z), 12)) # rounded, so every computer gives the same digits
def forward(x):
h = [round(math.tanh(sum(w * xi for w, xi in zip(row, x)) + b), 12) for row, b in zip(W1, b1)]
return h, sigmoid(sum(w * hi for w, hi in zip(W2, h)) + b2)
for epoch in range(1, 1501):
total = 0.0
for x, y in DATA:
h, p = forward(x)
total += (p - y) ** 2
dz2 = 2 * (p - y) * p * (1 - p)
dh = [dz2 * w for w in W2]
for j in range(HIDDEN):
W2[j] -= RATE * dz2 * h[j]
dz1 = dh[j] * (1 - h[j] ** 2)
for i in range(2):
W1[j][i] -= RATE * dz1 * x[i]
b1[j] -= RATE * dz1
b2 -= RATE * dz2
if epoch in (1, 100, 500, 1500):
print(f"epoch {epoch:4}: loss {total / 4:.4f}")
for x, y in DATA:
print(f"{x} -> {forward(x)[1]:.2f} (target {y:.0f})")Line by line
- The four XOR examples are the whole dataset; the network has 2 inputs, 3 tanh hidden neurons and a sigmoid output.
- Weights start small and random, with a fixed seed so the run is repeatable.
- Each example's forward pass is followed by the backward pass written out by hand, layer by layer, and an immediate update.
- The first epoch's loss, about 0.32, is what a network that knows nothing gets; it falls as the hidden neurons find useful features.
- After 1,500 epochs the outputs are close to 0 for 00 and 11 and close to 1 for 01 and 10: the network has learned XOR.
Expected result
epoch 1: loss 0.3218
epoch 100: loss 0.0426
epoch 500: loss 0.0010
epoch 1500: loss 0.0003
[0.0, 0.0] -> 0.01 (target 0)
[0.0, 1.0] -> 0.98 (target 1)
[1.0, 0.0] -> 0.98 (target 1)
[1.0, 1.0] -> 0.02 (target 0)
In the real world
MLPs are still everywhere: inside every transformer layer, in recommendation systems that combine user and item features, and in tabular-data models. The training loop in this lesson, with mini-batches and a better optimizer, is the same loop that trains models with billions of parameters.
Common mistakes
- Updating W2 before computing the gradient that flows into the hidden layer, which then uses the new weights.
- Forgetting the derivative of the hidden activation (here 1 − h²).
- Starting every weight at the same value, so every hidden neuron learns the same thing.
- Judging training by one epoch; look at the trend of the loss.
Best practices
- Count parameters before training; it tells you how big the model really is.
- Fix the random seed while developing, so runs can be compared.
- Print the loss every so often and stop when it stops improving.
Summary
A hidden layer lets a network learn features and solve problems such as XOR that a single neuron cannot. Training loops over epochs and batches: forward pass, loss, backward pass, update. The parameter count of a layer is inputs × neurons + neurons.
Quick check
Key takeaways
- A hidden layer lets a network build new features and draw curved boundaries.
- Training repeats forward, loss, backward and update, epoch after epoch.
- A layer with n inputs and m neurons has n × m weights and m biases.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.