A tiny language model

The problem

Train a character-level model on a short text and use it to generate new text, first with a simple next-character model and then with a small attention layer.

Requirements

Technologies

Step by step

  1. Choose a short text, such as a few paragraphs, and split it into training and validation parts.
  2. Train and evaluate the bigram model; generate a few lines from it.
  3. Implement attention over the last few characters with a causal mask, and check it on a tiny example by hand.
  4. Train the attention model and compare losses with the baseline.
  5. Implement the three decoding methods and compare their outputs.

Expected result

A small language model that produces text in the style of its training data, with an analysis of what attention adds.

How it is evaluated

Ways to improve it