A question-answering assistant over your notes
The problem
Build a small retrieval-augmented assistant in plain Python that answers questions from your own notes, cites its sources, refuses when the notes do not contain the answer, and is tested on questions you wrote.
Requirements
- Write at least 20 short notes, cut them into overlapping chunks and keep each chunk's source
- Rank chunks for a question with a word-overlap score and with averaged embeddings you define by hand, and compare the two
- Build a grounded prompt with numbered sources and an instruction to say when the sources do not contain the answer
- Use a simple extractive function as the stand-in model: it returns the best sentence and its citation, or says it does not know
- Test on at least 10 questions with known answers, report exact match and unigram recall, and add a guard that flags answers with weak support
Technologies
- Python
- Chunking
- Cosine similarity
- RAG prompts
- Evaluation
Step by step
- Write the notes and the test questions with their right answers.
- Write the chunking function and check it on one note by hand.
- Write the two retrievers and compare which finds the right chunk for more questions.
- Build the prompt, the stand-in model and the support check.
- Run the whole test set and write a short report of the scores and the failures.
Expected result
A working assistant with a table of scores and a short explanation of why it fails where it fails.
How it is evaluated
- Every answer carries a citation to a real chunk
- Questions without an answer in the notes get a refusal, not a guess
- The two retrievers are compared on the same questions
- The report lists at least three failures and their causes
Ways to improve it
- Add a re-ranking step that prefers chunks with more of the question words in a row
- Add a redaction step for emails and phone numbers
- Replace the stand-in model with a call to a library, shown as text only