A tiny language model
The problem
Train a character-level model on a short text and use it to generate new text, first with a simple next-character model and then with a small attention layer.
Requirements
- Turn a text into characters and build an embedding table for them
- Train a bigram baseline that predicts the next character from the current one
- Add a single causal self-attention layer with positional encodings and a feed-forward step
- Train with cross-entropy and compare the validation loss with the bigram baseline
- Generate text with greedy decoding, beam search and sampling with a temperature
Technologies
- Python
- Embeddings
- Attention
- Cross-entropy
- Decoding
Step by step
- Choose a short text, such as a few paragraphs, and split it into training and validation parts.
- Train and evaluate the bigram model; generate a few lines from it.
- Implement attention over the last few characters with a causal mask, and check it on a tiny example by hand.
- Train the attention model and compare losses with the baseline.
- Implement the three decoding methods and compare their outputs.
Expected result
A small language model that produces text in the style of its training data, with an analysis of what attention adds.
How it is evaluated
- The attention model reaches a lower validation loss than the bigram model
- The causal mask is tested: no position uses later characters
- Generated samples and losses are reported for each decoding method
- Every component has a small test with known results
Ways to improve it
- Add a second attention head and a second block
- Train on word pieces instead of single characters
- Plot attention weights for a sample sentence