Introduction
Blurring a photo, sharpening it, finding its edges, and even the first layer of a neural network that recognises cats all use one operation: slide a small grid of numbers over the image and, at each position, add up weight × pixel. That small grid is a kernel, and the operation is convolution (or its twin, correlation). Once you can compute it by hand, most of classic image filtering is just a choice of kernel.
How it works
Take a 3 × 3 kernel K and place its centre on pixel (r, c). Multiply each kernel weight by the pixel underneath it and add the nine products. That sum is the output at (r, c). Then move one pixel to the right and repeat, row by row:
out(r, c) = Σᵢ Σⱼ K[i][j] × img[r + i][c + j]
(Here the window's top-left corner is at (r, c), so the output is centred on pixel (r + 1, c + 1).) This is correlation. Convolution, the version from mathematics, first flips the kernel top to bottom and left to right. For symmetric kernels such as a blur the two are identical; for an edge kernel the flip changes the sign of the result.
Some useful kernels:
- Identity: 1 in the centre and 0 elsewhere. Output equals input.
- Box blur: nine 1s, then divide by 9. Each pixel becomes the mean of its neighbourhood.
- Sharpen: 5 in the centre and −1 above, below, left and right. It adds back the difference from the neighbours, boosting edges.
- Vertical edge: columns −1, 0, 1. It measures how much brighter the right side is than the left.
Output size. A k × k kernel fits completely inside an H × W image at (H − k + 1) × (W − k + 1) positions; this is the valid output. To keep the same size, pad the image with k // 2 extra pixels on each side. What goes in the padding is a choice:
- zero: pretend the outside is black;
- replicate: repeat the edge pixel (aaa|abc|ccc);
- reflect: mirror the image without repeating the edge pixel (cb|abc|ba), OpenCV's default.
Zero padding darkens the borders of a blurred image; replicate and reflect avoid that.
Normalising. If the weights add up to 1 (or you divide by their sum), a flat area keeps its brightness. Edge kernels add up to 0, so flat areas give 0 and only changes produce a response; their results can be negative, so clamp or take absolute values before showing them.
Example
def correlate(img, kernel):
k = len(kernel)
out = []
for r in range(len(img) - k + 1):
out.append([sum([kernel[i][j] * img[r + i][c + j] for i in range(k) for j in range(k)]) for c in range(len(img[0]) - k + 1)])
return out
img = [
[20, 20, 200, 200, 200],
[20, 20, 200, 200, 200],
[20, 20, 200, 200, 200],
[20, 20, 200, 200, 200],
]
box = [[1, 1, 1], [1, 1, 1], [1, 1, 1]]
edge = [[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]]
blurred = [[v // 9 for v in row] for row in correlate(img, box)]
print("box blur (sum // 9):", blurred)
print("vertical-edge kernel:", correlate(img, edge))
print("output size:", len(blurred), "x", len(blurred[0]), "from", len(img), "x", len(img[0]))
flipped = [row[::-1] for row in edge[::-1]]
print("edge kernel flipped for true convolution:", flipped)
print("convolution with it:", correlate(img, flipped)[0])Line by line
correlateslides a k × k kernel over every position where it fits and sums kernel × pixel.- The 4 × 5 image has a dark band (20) on the left and a bright area (200) on the right, with the step between columns 1 and 2.
- The box kernel sums nine pixels; dividing by 9 gives the mean. Windows over the step give 80 and 140, so the sharp step becomes a gradual ramp: that is blurring.
- The edge kernel subtracts the left column from the right column. Where the window crosses the step it gives (200 − 20) × 3 = 540; in the flat bright area it gives 0.
- Valid output from a 4 × 5 image with a 3 × 3 kernel is 2 × 3.
- Flipping the edge kernel turns −1, 0, 1 into 1, 0, −1, so true convolution gives −540: the same edge, opposite sign.
Expected result
box blur (sum // 9): [[80, 140, 200], [80, 140, 200]]
vertical-edge kernel: [[540, 540, 0], [540, 540, 0]]
output size: 2 x 3 from 4 x 5
edge kernel flipped for true convolution: [[1, 0, -1], [1, 0, -1], [1, 0, -1]]
convolution with it: [-540, -540, 0]
In the real world
Photo apps implement "sharpen", "soften" and "emboss" as kernels, and graphics cards apply them to millions of pixels at once. In OpenCV (shown as text; OpenCV does not run on this site), filter2D computes correlation and reflects at the borders by default:
import cv2
import numpy as np
sharpen = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]], dtype=np.float32)
out = cv2.filter2D(img, -1, sharpen) # same size, reflect borders
out_zero = cv2.filter2D(img, -1, sharpen, borderType=cv2.BORDER_CONSTANT)
Convolutional neural networks also compute correlation, but call it convolution. Because their kernels are learned, the flip makes no difference to what they can do.
Common mistakes
- Writing into the image you are reading. Later windows then see already-filtered pixels. Always write to a new image.
- Forgetting to divide a blur by the sum of its weights, which makes the image much brighter.
- Ignoring negative results from edge kernels, or letting values go past 255 without clamping.
- Assuming correlation and convolution are the same for a kernel that is not symmetric.
- Getting the output size wrong at the borders, which shifts the result by one pixel.
Best practices
- Test a new kernel on a tiny image with a single bright pixel: the output shows the kernel itself (flipped, for convolution).
- Choose the border mode on purpose; replicate or reflect usually look best.
- Keep results in a wider type (or Python integers) and clamp only at the end.
- Check that a blur kernel's weights add up to 1 and an edge kernel's add up to 0.
Summary
Filtering slides a kernel over the image and replaces each pixel with a weighted sum of its neighbourhood. A k × k kernel gives (H − k + 1) × (W − k + 1) valid outputs, and padding by k // 2 keeps the size, filling the border with zeros, copies or mirrors. Correlation uses the kernel as written; convolution flips it first, which matters only for kernels that are not symmetric.
Quick check
Key takeaways
- Every output pixel is a weighted sum of the input pixels under the kernel.
- A k × k kernel on an H × W image gives (H − k + 1) × (W − k + 1) valid outputs; padding by k // 2 keeps the size.
- Libraries such as OpenCV's filter2D compute correlation; true convolution flips the kernel, which matters only for kernels that are not symmetric.
Practice problems for this lesson
My notes
Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.