Find out where a transformer's parameters are.
What to build
A program that counts the parameters of a GPT-style transformer.
Requirements
- The model has a token embedding table (V × d), learned position embeddings (L × d), N blocks and a final layer norm (2d). Each block has attention 4(d² + d), a feed-forward network d·f + f + f·d + d and two layer norms (2 × 2d). The output layer reuses the token embedding table.
- Print
embeddings E,positions P,per block B (attention A, feed-forward F, layer norms N),all blocks T,final layer norm Mandtotal S.
Input
One line: V, L, d, f and N.
Output
Six lines.
Constraints
All values below 100,000.
Example
Input:
10 4 2 8 1
Output:
embeddings 20
positions 8
per block 74 (attention 24, feed-forward 42, layer norms 8)
all blocks 74
final layer norm 4
total 106
Why it works
Adding up the matrices and bias vectors component by component gives the exact size of the model and shows which parts dominate.
Examples
Example 1 · Input
10 4 2 8 1
Expected output
embeddings 20 positions 8 per block 74 (attention 24, feed-forward 42, layer norms 8) all blocks 74 final layer norm 4 total 106
Example 2 · Input
50257 1024 768 3072 12
Expected output
embeddings 38597376 positions 786432 per block 7087872 (attention 2362368, feed-forward 4722432, layer norms 3072) all blocks 85054464 final layer norm 1536 total 124439808
Prompts inside input() are ignored when checking, so input("n: ") is fine.
Hints
Hint 1
Attention has four d × d matrices (query, key, value, output), each with d biases: 4(d² + d).
Hint 2
Feed-forward: d × f + f + f × d + d. Each layer norm has 2d parameters (scale and shift); a block has two.
Solution
Show one correct solution (try it yourself first)
V, L, d, f, N = map(int, input().split())
embeddings = V * d
positions = L * d
attention = 4 * (d * d + d)
feed_forward = d * f + f + f * d + d
norms = 2 * 2 * d
block = attention + feed_forward + norms
final = 2 * d
print(f"embeddings {embeddings}")
print(f"positions {positions}")
print(f"per block {block} (attention {attention}, feed-forward {feed_forward}, layer norms {norms})")
print(f"all blocks {N * block}")
print(f"final layer norm {final}")
print(f"total {embeddings + positions + N * block + final}")