Convolutional neural networks for images

Advanced

Introduction

In the last lesson you designed features by hand: gradient histograms that capture which way edges run. That works, but someone must decide what to measure. Convolutional neural networks (CNNs) learn the features instead. Their first layers end up detecting edges and colour blobs, later layers combine those into textures and parts, and the last layers recognise whole objects. Since AlexNet won the ImageNet competition in 2012 by a wide margin, CNNs and their successors have led almost every vision task. This lesson shows what happens inside one, using operations you already know.

How it works

Convolution layers. An input has C channels (3 for a colour image). A filter is a stack of C kernels, one per channel, plus a bias. At each position it multiplies and adds over all channels and all kernel cells, giving one number; over all positions it gives one feature map. A layer with F filters outputs F channels. With kernel size K, stride S and padding P, the output height is

H_out = (H + 2P − K) // S + 1

and the same for the width. A layer has F × (K × K × C + 1) parameters (the +1 is each filter's bias), no matter how big the image is. Compare a dense layer from a 224 × 224 × 3 image to just 100 units: over 15 million weights. Weight sharing, using one kernel everywhere, is what makes CNNs practical, and it means a pattern learned in one place is recognised anywhere.

ReLU. After each convolution, the activation max(0, z) keeps positive responses and zeroes the rest, so a feature map marks only where its pattern is present. Without a non-linear activation, stacked convolutions would collapse into one big convolution.

Pooling. Max pooling with 2 × 2 blocks and stride 2 keeps the strongest response in each block, halving the height and width. Later layers then see a wider area for the same cost, and a pattern that moves by a pixel usually gives the same pooled value.

Receptive field. Each value in layer two depends on a 3 × 3 patch of layer one, which depends on a 5 × 5 patch of the image. Stacking layers and pooling grows the receptive field, so deep layers can respond to whole objects.

The classifier. After several conv-ReLU-pool stages, the final feature maps are flattened and fed to dense layers ending in a softmax over the classes. The whole network is trained end to end with backpropagation and gradient descent, as in the Deep Learning course; the kernels start random and become edge, texture and part detectors because that reduces the loss.

Example

img = [
    [0, 0, 9, 0, 0, 0],
    [0, 0, 9, 0, 0, 0],
    [0, 0, 9, 0, 0, 0],
    [0, 0, 9, 0, 0, 0],
    [0, 0, 9, 0, 0, 0],
    [0, 0, 9, 0, 0, 0],
]
kernels = [
    ("vertical", [[-1, 2, -1], [-1, 2, -1], [-1, 2, -1]]),
    ("horizontal", [[-1, -1, -1], [2, 2, 2], [-1, -1, -1]]),
]


def conv_relu(img, k):
    n = len(img) - 2
    return [[max(0, sum([k[i][0] * img[r + i][c] + k[i][1] * img[r + i][c + 1] + k[i][2] * img[r + i][c + 2] for i in range(3)])) for c in range(n)] for r in range(n)]


def max_pool(f):
    n = len(f) // 2
    return [[max(f[2 * r][2 * c], f[2 * r][2 * c + 1], f[2 * r + 1][2 * c], f[2 * r + 1][2 * c + 1]) for c in range(n)] for r in range(n)]


scores = []
for name, k in kernels:
    fmap = conv_relu(img, k)
    pooled = max_pool(fmap)
    print(f"{name} filter after ReLU: {fmap}")
    print(f"   after 2 x 2 max pooling: {pooled}")
    scores.append((sum([sum(row) for row in pooled]), name))
print("scores:", scores, "-> a", max(scores)[1], "line")
c_in, c_out, k = 3, 16, 3
print(f"Conv2d({c_in}, {c_out}, {k}) has {c_out} x ({c_in} x {k} x {k} + 1) = {c_out * (c_in * k * k + 1)} parameters")
Open in compiler

Line by line

  1. The 6 × 6 image contains a vertical line in column 2.
  2. Two hand-set filters stand in for learned ones: one responds to vertical lines (−1, 2, −1 across), the other to horizontal lines (the same pattern turned on its side). Each sums to 0, so flat areas give 0.
  3. conv_relu computes the valid convolution, one kernel row at a time, and applies ReLU. The vertical filter gives 54 (2 × 9 × 3) where the line sits in the middle of the window; the negative responses beside the line become 0.
  4. The horizontal filter gives 0 everywhere: each of its columns adds −1 + 2 − 1 = 0 times the same value.
  5. max_pool turns each 4 × 4 map into 2 × 2 by keeping the largest value of each block.
  6. Summing each pooled map gives a score per filter; the vertical filter wins, so the network calls it a vertical line.
  7. The last line counts the parameters of a real layer with 3 input channels, 16 filters and 3 × 3 kernels: 16 × (27 + 1) = 448.

Expected result

vertical filter after ReLU: [[0, 54, 0, 0], [0, 54, 0, 0], [0, 54, 0, 0], [0, 54, 0, 0]]
   after 2 x 2 max pooling: [[54, 0], [54, 0]]
horizontal filter after ReLU: [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
   after 2 x 2 max pooling: [[0, 0], [0, 0]]
scores: [(108, 'vertical'), (0, 'horizontal')] -> a vertical line
Conv2d(3, 16, 3) has 16 x (3 x 3 x 3 + 1) = 448 parameters

In the real world

CNNs sort photos, read handwriting, inspect products, help radiologists and run in phone cameras. Famous designs include LeNet (1998) for digits, AlexNet (2012), VGG, and ResNet (2015), whose skip connections allow networks with over a hundred layers. Today vision transformers compete with CNNs, but convolution remains a core building block. Most projects start from a network pre-trained on a large dataset and fine-tune it (transfer learning). In PyTorch (shown as text; PyTorch does not run on this site):

import torch
from torch import nn

model = nn.Sequential(
    nn.Conv2d(1, 8, kernel_size=3, padding=1),    # 28 x 28 x 1  -> 28 x 28 x 8
    nn.ReLU(),
    nn.MaxPool2d(2),                              # -> 14 x 14 x 8
    nn.Conv2d(8, 16, kernel_size=3, padding=1),   # -> 14 x 14 x 16
    nn.ReLU(),
    nn.MaxPool2d(2),                              # -> 7 x 7 x 16
    nn.Flatten(),
    nn.Linear(16 * 7 * 7, 10),
)
x = torch.zeros(1, 1, 28, 28)                     # batch, channels, height, width
print(model(x).shape)                             # torch.Size([1, 10])

This model has 9,098 parameters: 80 and 1,168 in the convolutions and 7,850 in the final layer.

Common mistakes

  • Getting the channel order wrong: PyTorch expects (batch, channels, height, width), while images load as (height, width, channels).
  • Miscounting output sizes; use (H + 2P − K) // S + 1 at every layer and check with a dummy input.
  • Forgetting the bias when counting parameters.
  • Training a large CNN from scratch on a few hundred images, which overfits; fine-tune a pre-trained model instead.
  • Normalising training and test images differently.

Best practices

  • Write the shape after every layer as a comment, as in the example above.
  • Use small kernels (3 × 3) stacked deeply rather than large kernels.
  • Augment training images with flips, crops and small rotations to teach invariance.
  • Start from pre-trained weights whenever a similar task exists.

Summary

A CNN stacks convolution layers (learned filters spanning all input channels), ReLU and pooling, then classifies with dense layers, and training learns every kernel. Shapes follow (H + 2P − K) // S + 1, and a layer has F × (K × K × C + 1) parameters regardless of image size thanks to weight sharing. Pooling and depth grow the receptive field, letting later layers recognise whole objects.

Quick check

1. A 32 x 32 x 3 input goes through Conv2d with 16 filters, kernel 3, stride 1, padding 1. What is the output shape?
2. How many parameters does a conv layer with 8 filters of 5 x 5 on 1 input channel have?
3. What does weight sharing mean in a CNN?
4. What does 2 x 2 max pooling with stride 2 do to a 10 x 10 feature map?

Key takeaways

  • A CNN is a stack of convolution, ReLU and pooling layers followed by a classifier; training learns the kernels.
  • Output size is (H + 2P − K) // S + 1, and a layer has F × (K × K × C + 1) parameters, independent of the image size.
  • Weight sharing and pooling make CNNs efficient and tolerant of position, which is why they beat hand-designed features.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.