Meaning and modern NLP
Word embeddings, similarity and analogies, byte-pair-encoding tokenizers, and attention, transformers and how large language models generate text and are used.
About 4 hours of study
Learning objectives
- Compare and combine word vectors
- Train and apply a BPE tokenizer
- Compute attention and control LLM decoding
Lessons
- 1
Word embeddings Intermediate
Represent words as short dense vectors whose directions carry meaning. Compare words with cosine similarity, find nearest neighbours, build vectors from co-occurrence counts, and solve analogies such as king − man + woman with vector arithmetic.
- 2
Byte-pair encoding tokenization Intermediate
Learn how modern language models split text into sub-word tokens. Train byte-pair encoding by repeatedly merging the most frequent pair of symbols, then apply the learned merges to tokenize new words, and see how vocabulary size, unknown words and token counts trade off.
- 3
Attention, transformers and LLMs in practice Advanced
See how self-attention lets every token look at every other token, how transformers stack attention into large language models, and how an LLM turns scores into text with softmax, temperature, greedy, top-k and top-p decoding. Ends with how LLMs are used in practice.
Module practice
- NLP: Compare word vectorsEasy
- NLP: Vectors from co-occurrence countsMedium
- NLP: Solve word analogiesHard
- NLP: Count adjacent symbol pairsEasy
- NLP: Learn BPE mergesMedium
- NLP: Encode words with BPE mergesHard
- NLP: Softmax with temperatureEasy
- NLP: Greedy, top-k and top-pMedium
- NLP: Self-attention over a sentenceHard