Recurrent networks and LSTMs

Advanced

Introduction

Text, speech, music, sensor readings and stock prices are sequences: their meaning depends on order. A dense network has a fixed number of inputs and no notion of order. A recurrent neural network reads one element at a time and keeps a hidden state, a running summary of what it has seen. Recurrent networks, and especially LSTMs, powered translation, speech recognition and handwriting recognition before transformers.

How it works

At each step t, a simple RNN combines the new input with the previous hidden state:

hₜ = tanh(Wₓ xₜ + Wₕ hₜ₋₁ + b)

with h₀ = 0. The same weights are used at every step, so the network handles sequences of any length. An output can be read from the hidden state at every step (tagging each word) or only at the end (classifying a whole sentence).

Training unrolls the network over the sequence, which makes it an ordinary deep network with shared weights, and runs backpropagation through it: backpropagation through time. Because the weights are shared, each weight's gradient is the sum of its gradients at every step. Going back one step multiplies the gradient by Wₕ and by the tanh slope (1 − h²).

That repeated multiplication is the problem. If |Wₕ × slope| is below 1, the gradient from a step far in the past vanishes; above 1, it explodes. Plain RNNs therefore struggle to learn dependencies more than a few dozen steps long. Gradient clipping handles explosions; the cure for vanishing is a better cell.

An LSTM keeps a separate cell state c and uses gates, each a sigmoid of the input and previous hidden state:

  • forget gate f: how much of the old cell state to keep;
  • input gate i and candidate g = tanh(…): what new information to add;
  • output gate o: how much of the cell to reveal.

cₜ = f · cₜ₋₁ + i · g, hₜ = o · tanh(cₜ)

When f is near 1, the cell state, and its gradient, pass through almost unchanged, so information can survive hundreds of steps. The GRU is a simpler variant with two gates.

Syntax

h = 0.0
for x in sequence:
    h = math.tanh(wx * x + wh * h + b)

f, i, o = sigmoid(zf), sigmoid(zi), sigmoid(zo)    # LSTM gates
g = math.tanh(zg)
c = f * c + i * g
h = o * math.tanh(c)

Example

import math


def rnn(xs, wx, wh, b, h=0.0):
    states = []
    for x in xs:
        h = round(math.tanh(wx * x + wh * h + b), 12)  # rounded, so every computer gives the same digits
        states.append(h)
    return states


print("hidden states for the input 1, 0, 0, 0, 0, 0:")
for wh in [0.5, 0.9, 1.5]:
    states = rnn([1, 0, 0, 0, 0, 0], wx=1.0, wh=wh, b=0.0)
    print(f"    wh = {wh}: " + " ".join(f"{h:.3f}" for h in states))

print("gradient of the last state with respect to the first input:")
for wh in [0.5, 0.9]:
    for length in [5, 20]:
        states = rnn([1] + [0] * (length - 1), 1.0, wh, 0.0)
        grad = 1 - states[0] ** 2
        for h in states[1:]:
            grad *= wh * (1 - h ** 2)
        print(f"    wh = {wh}, {length} steps: {grad:.2e}")
Open in compiler

Line by line

  1. rnn applies the update to each input in turn and records the hidden states.
  2. With the input 1 followed by zeros, the first state remembers the 1; later states show how quickly the memory fades.
  3. With wₕ = 0.5 the memory halves each step; with 1.5 it is amplified at first, then tanh holds it near a fixed level.
  4. The gradient of the last state with respect to the first input multiplies wₕ and a tanh slope for every step.
  5. Over 20 steps it shrinks to almost nothing, especially with the smaller weight: the vanishing-gradient problem.

Expected result

hidden states for the input 1, 0, 0, 0, 0, 0:
    wh = 0.5: 0.762 0.363 0.180 0.090 0.045 0.022
    wh = 0.9: 0.762 0.595 0.490 0.414 0.356 0.310
    wh = 1.5: 0.762 0.815 0.840 0.851 0.856 0.857
gradient of the last state with respect to the first input:
    wh = 0.5, 5 steps: 2.18e-02
    wh = 0.5, 20 steps: 6.66e-07
    wh = 0.9, 5 steps: 9.79e-02
    wh = 0.9, 20 steps: 1.29e-02

In the real world

LSTMs ran Google Translate and the speech recognizers of phone assistants in the mid-2010s, and they still appear in time-series forecasting and on small devices. Transformers have replaced them for most language tasks because attention links distant positions directly, but some new sequence models revisit recurrent ideas to process very long inputs efficiently.

Common mistakes

  • Using different weights at each step, which breaks the idea of a recurrent network.
  • Overwriting a shared weight's gradient at each step instead of adding to it.
  • Forgetting to reset the hidden state between unrelated sequences.
  • Training a plain RNN on very long dependencies without clipping or gated cells.

Best practices

  • Use LSTM or GRU cells rather than plain RNNs for anything beyond short sequences.
  • Clip gradients to a maximum norm when training recurrent models.
  • Check shapes over time: one hidden state per step, the same weights throughout.

Summary

An RNN reads a sequence step by step, updating a hidden state with shared weights. Backpropagation through time adds each weight's gradients over all steps, but repeated multiplication makes long-range gradients vanish or explode. LSTMs keep a gated cell state that carries information and gradients across many steps.

Quick check

1. What does an RNN carry from one step to the next?
2. Why are an RNN's weight gradients summed over time steps?
3. What causes vanishing gradients in a plain RNN?
4. What does an LSTM's forget gate control?

Key takeaways

  • The same weights are used at every time step, so their gradients add up over the steps.
  • Repeated multiplication by the recurrent weight and tanh slopes makes long-range gradients vanish or explode.
  • LSTM gates let information and gradients flow through the cell state almost unchanged for many steps.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.