Meaning and modern NLP

Word embeddings, similarity and analogies, byte-pair-encoding tokenizers, and attention, transformers and how large language models generate text and are used.

About 4 hours of study

Learning objectives

Lessons

  1. 1

    Word embeddings Intermediate

    Represent words as short dense vectors whose directions carry meaning. Compare words with cosine similarity, find nearest neighbours, build vectors from co-occurrence counts, and solve analogies such as king − man + woman with vector arithmetic.

  2. 2

    Byte-pair encoding tokenization Intermediate

    Learn how modern language models split text into sub-word tokens. Train byte-pair encoding by repeatedly merging the most frequent pair of symbols, then apply the learned merges to tokenize new words, and see how vocabulary size, unknown words and token counts trade off.

  3. 3

    Attention, transformers and LLMs in practice Advanced

    See how self-attention lets every token look at every other token, how transformers stack attention into large language models, and how an LLM turns scores into text with softmax, temperature, greedy, top-k and top-p decoding. Ends with how LLMs are used in practice.

Module practice

Module quiz

1. What does cosine similarity ignore?
2. What does a BPE tokenizer do with a rare word?
3. Words used in similar contexts get similar co-occurrence vectors. Which idea is this?
4. Why does a transformer need positional information?
5. What is top-k sampling with k = 1 the same as?
6. What is retrieval-augmented generation?

‹ Structure in textFinal assessment ›