Neurons, layers and activation functions

Intermediate

Introduction

Deep learning powers image recognition, speech assistants, translation and chatbots. Underneath, every one of those systems is built from one simple unit repeated millions of times: the artificial neuron. This course builds deep learning from that unit upwards in plain Python, with no libraries, so that nothing is hidden: by the end you will have trained networks, written backpropagation, convolution, recurrent cells and attention yourself.

How it works

A neuron receives inputs x₁ … xₙ. Each input has a weight wᵢ, and the neuron has a bias b. It computes the weighted sum

z = w₁x₁ + w₂x₂ + … + wₙxₙ + b

and then its output a = f(z), where f is the activation function. The weights say how much each input matters, the bias shifts the point at which the neuron responds, and the activation decides the shape of the response.

Common activations:

  • sigmoid, 1 / (1 + e^(−z)): squashes any number into (0, 1); useful for probabilities.
  • tanh: squashes into (−1, 1) and is centred on 0.
  • ReLU, max(0, z): passes positive values, blocks negative ones. It is cheap and trains well, so it is the usual choice for hidden layers.

The activation must be non-linear. A linear function of a linear function is still linear, so a deep stack of linear layers could only ever draw straight lines through the data. Non-linear activations let networks represent curves, corners and complicated decision boundaries.

A layer is several neurons reading the same inputs. Stacking their weight vectors as the rows of a matrix W, the whole layer is a = f(W x + b): one matrix-vector product, one vector addition and one activation applied element by element. A network chains layers, each reading the previous layer's outputs.

Syntax

def dense(x, W, b, f):
    return [f(sum(w * xi for w, xi in zip(row, x)) + bi) for row, bi in zip(W, b)]

Example

import math


def sigmoid(z):
    return 1 / (1 + round(math.exp(-z), 12))  # rounded, so every computer gives the same digits


def relu(z):
    return max(0.0, z)


def neuron(inputs, weights, bias, activation):
    z = sum(x * w for x, w in zip(inputs, weights)) + bias
    return z, activation(z)


inputs = [1.0, 2.0, -1.0]
weights = [0.5, -0.25, 1.0]
bias = 0.5
for name, f in [("sigmoid", sigmoid), ("tanh", lambda z: round(math.tanh(z), 12)), ("ReLU", relu)]:
    z, a = neuron(inputs, weights, bias, f)
    print(f"{name:8} z = {z:.2f}, output = {a:.4f}")

W = [[0.2, -0.5, 1.0], [1.5, 0.0, -2.0], [-1.0, 1.0, 0.5]]
b = [0.1, -0.2, 0.0]
layer = [relu(sum(w * x for w, x in zip(row, inputs)) + b_i) for row, b_i in zip(W, b)]
print("a layer of three ReLU neurons:", [round(v, 2) for v in layer])
Open in compiler

Line by line

  1. neuron computes the weighted sum z and then applies the activation.
  2. For these inputs and weights, z = 0.5 − 0.5 − 1 + 0.5 = −0.5.
  3. Sigmoid turns −0.5 into 0.3775, tanh into −0.4621, and ReLU blocks it completely, giving 0.
  4. The layer has three neurons; each row of W is one neuron's weights and b holds their biases.
  5. The first neuron's sum is negative, so ReLU outputs 0; the others pass 3.3 and 0.5.

Expected result

sigmoid  z = -0.50, output = 0.3775
tanh     z = -0.50, output = -0.4621
ReLU     z = -0.50, output = 0.0000
a layer of three ReLU neurons: [0.0, 3.3, 0.5]

In the real world

Large language models contain billions of weights, but each layer still computes W x + b followed by an activation, on graphics processors that do millions of these products in parallel. Variants of ReLU, such as GELU, are used in transformers. Sigmoid is still the standard output for yes/no predictions such as "is this email spam?".

Common mistakes

  • Forgetting the bias, which forces every neuron's boundary through the origin.
  • Using no activation in hidden layers, which makes the whole network linear.
  • Mixing up rows and columns of the weight matrix.
  • Using sigmoid in every layer of a deep network, which makes training slow (the gradients become tiny).

Best practices

  • Use ReLU or a variant in hidden layers, and choose the output activation to match the task.
  • Check the shapes: W has one row per neuron and one column per input.
  • Compute one small example by hand before trusting the code.

Summary

A neuron computes a weighted sum of its inputs plus a bias and passes it through a non-linear activation. Sigmoid and tanh squash values into a range; ReLU keeps positive values and zeroes negative ones. A layer applies many neurons at once as a matrix-vector product, and a network chains layers.

Quick check

1. What does a neuron compute before its activation?
2. What is ReLU(−2)?
3. Why must activations be non-linear?
4. Which activation squashes values into (0, 1)?

Key takeaways

  • A neuron is a dot product, a bias and a non-linear function.
  • Without non-linear activations, any stack of layers collapses into one linear function.
  • ReLU, max(0, z), is the default for hidden layers; sigmoid and tanh squash values into a range.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.