Testing a document retrieval assistant before it meets a customer
Retrieval-augmented assistants are easy to demo and harder to trust. A practical way to decide what "good enough" means before launch, using questions your own team writes.
An assistant that answers questions from your own documents, usually called retrieval-augmented generation or RAG, can look impressive within a day. It finds a paragraph, a language model rephrases it, and the answer sounds confident. The difficulty is that confidence and correctness are not the same thing, and a demo only ever shows the questions somebody expected.
Before an assistant like this is put in front of customers or staff, it needs a test that reflects the questions people will actually ask. This note describes how we approach that.
Start with an evaluation set, not a prompt
An evaluation set is a list of real questions with the answer you would accept for each, and the document that answer should come from. It is written by people who know the material: support staff, operations leads, the author of the policy. Aim for variety rather than volume: common questions, awkward phrasings, questions whose answer changed recently, and questions the assistant should decline because the documents do not cover them.
Measure retrieval and answering separately
Most wrong answers begin with the wrong passage being retrieved. Checking the two stages separately tells you where to spend effort:
- Retrieval: for each question, did the right source appear among the passages handed to the model?
- Answering: given the right passage, did the assistant state the answer correctly and cite where it came from?
- Refusal: when the documents do not contain an answer, did it say so rather than guess?
Decide the thresholds before you look at results
Agree in advance what level of accuracy is acceptable for launch, and which kinds of mistake are unacceptable at any level, such as inventing a refund policy. Deciding afterwards makes it too easy to accept whatever the first run produced.
An assistant that says "I can't find that in our documents" is often more valuable than one that is right most of the time and confidently wrong the rest.
Keep the evaluation set after launch
Documents change. Re-running the same set of questions whenever the source material or the model changes is the simplest way to notice a regression before a customer does. Questions that users ask in production and the assistant gets wrong should be added to the set over time.
When not to build one
If the underlying documents are out of date or contradict each other, an assistant will faithfully repeat the contradiction. Sometimes the right first step is tidying the source material, or a smaller feasibility assessment that establishes whether the data can support the idea at all.
Working on something like this?
Send a short brief and we will reply in writing with the service that fits.
