Building with models and judging them

Prompt templates, validated JSON and tool loops, and the scores used to evaluate generated text.

About 5 hours of study

Learning objectives

Lessons

  1. 1

    Prompt engineering Intermediate

    Writing instructions a model can follow: the parts of a good prompt, zero-shot and few-shot prompting, templates with placeholders, system and user messages, and how to test a prompt.

  2. 2

    Structured output and tool calling Intermediate

    Making a model's answer usable by programs: asking for JSON, validating it, retrying on errors, and the tool-calling loop in which a model requests functions and your code runs them.

  3. 3

    Evaluating generative models Intermediate

    How to tell whether generated text is good: exact match, n-gram overlap scores such as BLEU and ROUGE, perplexity, and human or model-as-judge ratings, with the strengths and limits of each.

Module practice

Module quiz

1. Zero-shot prompting means giving...
2. In a tool-calling loop, who actually runs the tool?
3. Exact match counts an answer as right when...
4. The schema needs a name and an integer age, but the reply has only a name. Validation should report...
5. The candidate is the the the and the reference is the cat sat. What is the clipped unigram precision?
6. A candidate has 2 words, the reference 4, and every candidate n-gram matches. The brevity penalty is about...

‹ Embeddings, search and retrievalImages, tuning and safety ›