Start with the documents needed for the task. A tool asked to explain your delivery policy needs the current policy and any relevant exceptions. Giving it the whole shared drive adds places to find an obsolete answer.
Suppose your folder contains “Delivery terms final”, “Delivery terms final updated” and a PDF used last year. People may know which one applies. The tool needs a clearer arrangement.
Make a small source set
Here is an illustrative record for a trial that drafts answers to delivery questions:
Information | Agreed source | Who maintains it |
|---|---|---|
Standard delivery times | Current delivery policy, with its effective date | Operations manager |
Charges by area | Current delivery-charge list | Person responsible for pricing |
Exceptions already agreed with a customer | The relevant approved order record | Person handling the order |
Keep older copies outside the tool’s permitted source set. Retain them elsewhere if the business needs them as records. Ask whoever supports the system to check what the tool can actually read: a folder name alone does not restrict access.
If a customer exception exists only in an email or someone’s memory, the tool cannot reliably account for it. Record it in the approved source or leave that answer to a person. Agree which source takes precedence if the standard policy and the order record differ.
Test answers and source use separately
Ask a question whose answer you know, such as the delivery charge for a named area. Compare the response with the current list. This tests whether the answer is correct for that example.
If the tool supplies a document link or quotation, open it and check that it supports the answer. A source reference can itself be wrong. If the tool gives no inspectable source evidence, a correct answer alone does not establish which document it used. Record that limit instead of treating correctness as proof of retrieval.
Then ask about something the documents do not cover. Does the tool make the gap clear, or supply a plausible answer? Use clearly labelled fictional material in a separate test area if you need to test conflicting versions. Keep that material out of the working source set.
Keep the task small enough to check
A test on a few policy documents does not establish performance across a large archive. Add material only when the task needs it, and rerun relevant questions when an important document changes.
Someone needs to maintain the working set and remove superseded copies from the tool’s reach. That can be the person who already maintains the policy. Count the extra effort when judging whether the tool helps.
Before using customer material, settle the tool and information permissions. For the document work itself, the useful outcome is a source set whose contents and limitations you can explain.
About Jay
I’m Jay Janes, a Practical AI Advisor and founder of Reformat Labs. I write about choosing useful work for AI, checking the results and supporting the people using it.