Deep learning: Count a transformer's parameters

Find out where a transformer's parameters are.

What to build

A program that counts the parameters of a GPT-style transformer.

Requirements

  • The model has a token embedding table (V × d), learned position embeddings (L × d), N blocks and a final layer norm (2d). Each block has attention 4(d² + d), a feed-forward network d·f + f + f·d + d and two layer norms (2 × 2d). The output layer reuses the token embedding table.
  • Print embeddings E, positions P, per block B (attention A, feed-forward F, layer norms N), all blocks T, final layer norm M and total S.

Input

One line: V, L, d, f and N.

Output

Six lines.

Constraints

All values below 100,000.

Example

Input:

10 4 2 8 1

Output:

embeddings 20
positions 8
per block 74 (attention 24, feed-forward 42, layer norms 8)
all blocks 74
final layer norm 4
total 106

Why it works

Adding up the matrices and bias vectors component by component gives the exact size of the model and shows which parts dominate.

Examples

Example 1 · Input
10 4 2 8 1
Expected output
embeddings 20
positions 8
per block 74 (attention 24, feed-forward 42, layer norms 8)
all blocks 74
final layer norm 4
total 106
Example 2 · Input
50257 1024 768 3072 12
Expected output
embeddings 38597376
positions 786432
per block 7087872 (attention 2362368, feed-forward 4722432, layer norms 3072)
all blocks 85054464
final layer norm 1536
total 124439808

Prompts inside input() are ignored when checking, so input("n: ") is fine.

Hints

Hint 1

Attention has four d × d matrices (query, key, value, output), each with d biases: 4(d² + d).

Hint 2

Feed-forward: d × f + f + f × d + d. Each layer norm has 2d parameters (scale and shift); a block has two.

Solution

Show one correct solution (try it yourself first)
V, L, d, f, N = map(int, input().split())
embeddings = V * d
positions = L * d
attention = 4 * (d * d + d)
feed_forward = d * f + f + f * d + d
norms = 2 * 2 * d
block = attention + feed_forward + norms
final = 2 * d
print(f"embeddings {embeddings}")
print(f"positions {positions}")
print(f"per block {block} (attention {attention}, feed-forward {feed_forward}, layer norms {norms})")
print(f"all blocks {N * block}")
print(f"final layer norm {final}")
print(f"total {embeddings + positions + N * block + final}")

Lesson: Transformers →