Deep learning: One step of an LSTM

Compute the gates and states of an LSTM cell.

What to build

A program that computes one step of a one-unit LSTM.

Requirements

  • Gates are given in the order forget, input, candidate, output, each as wh wx b. f, i and o use sigmoid; the candidate g uses tanh.
  • Print f F, i I, g G, o O, c C and h H, each with four decimal places.

Input

Line 1: x, h_prev and c_prev. Then four lines, one per gate.

Output

Six lines.

Constraints

Values between −100 and 100.

Example

Input:

5 0 1
0 0 0
0 0 0
0 0 0
0 0 0

Output:

f 0.5000
i 0.5000
g 0.0000
o 0.5000
c 0.5000
h 0.2311

Why it works

The forget gate scales the old memory, the input gate scales the new candidate, and the output gate decides how much of the updated memory becomes the hidden state.

Examples

Example 1 · Input
5 0 1
0 0 0
0 0 0
0 0 0
0 0 0
Expected output
f 0.5000
i 0.5000
g 0.0000
o 0.5000
c 0.5000
h 0.2311
Example 2 · Input
1 0 0
0 0 0
0 1 0
0 1 0
0 2 0
Expected output
f 0.5000
i 0.7311
g 0.7616
o 0.8808
c 0.5568
h 0.4453

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Each gate has three numbers: a weight for h_prev, a weight for x and a bias; z = wh × h_prev + wx × x + b.

Hint 2

f, i, o use sigmoid and g uses tanh; then c = f × c_prev + i × g and h = o × tanh(c).

Solution

Show one correct solution (try it yourself first)
import math

x, h_prev, c_prev = map(float, input().split())


def sigmoid(z):
    return 1 / (1 + math.exp(-z))


gates = []
for _ in range(4):
    wh, wx, b = map(float, input().split())
    gates.append(wh * h_prev + wx * x + b)
f, i, o = sigmoid(gates[0]), sigmoid(gates[1]), sigmoid(gates[3])
g = math.tanh(gates[2])
c = f * c_prev + i * g
h = o * math.tanh(c)
for name, value in [("f", f), ("i", i), ("g", g), ("o", o), ("c", c), ("h", h)]:
    print(f"{name} {value + 0.0:.4f}")

Lesson: Recurrent networks and LSTMs →