A prompt, tool and evaluation test bench
The problem
Build a small test bench that fills prompt templates, runs a scripted model through a validated tool-calling loop, and scores the results, so you can see which prompt wording and which checks work best.
Requirements
- Write a template with placeholders and build zero-shot and few-shot prompts from it for at least 12 inputs
- Write a scripted stand-in model that gives a valid reply for some inputs and a broken one for others
- Validate every reply as JSON against a schema, retry up to three times with an error message, and log each attempt
- Add at least two tools and a loop that stops at a final answer or after a step limit
- Score the final answers with exact match and a clipped bigram precision, and report both for each prompt variant
Technologies
- Python
- Prompt templates
- JSON validation
- Tool calling
- Evaluation metrics
Step by step
- Write the template, the inputs and the right answers.
- Write the stand-in model so that its mistakes are easy to recognise.
- Write the validator, the retry loop and the tool loop.
- Score each prompt variant on the full set of inputs.
- Write a short note on which variant won and why you trust the result.
Expected result
A test bench that prints a score table for each prompt variant and a log of every rejected reply.
How it is evaluated
- No tool runs unless its name and arguments pass the checks
- Every loop ends because of a final answer, a retry limit or a step limit
- Both prompt variants are scored on the same inputs
- The note explains how many inputs you tested and why that is enough or not enough
Ways to improve it
- Add a rubric function that acts as a judge and measure how often it agrees with exact match
- Add a tool that must be approved before it runs
- Add a fixed seed and sampling to the stand-in model so mistakes happen at random