Teaching scenario: The fictional return policies below are created for this exercise. They are not policies of a real business or interpretations of legal requirements.
Relevance does not settle the decisionA customer asks whether an item can be returned after 20 days. Retrieval finds two passages. A general help article says returns are accepted within 30 days. A product-specific policy says the limit is 14 days. Both passages discuss the question, but their presence in the results does not establish which one applies to this purchase.
The system needs more than a plausible sentence and a citation. It needs enough context to determine whether the rule covers the product, transaction, customer, region, and relevant time. A newer document can describe a new policy without governing an older purchase.
Microsoft's RAG overview describes retrieval as grounding responses in external content and highlights the need for authorized, relevant information. That grounding is a starting point; deciding applicability still requires your application's rules and context. Primary reference: Microsoft RAG overview.
Build an evidence recordFor the exercise, capture the passages alongside their metadata. Do not ask the model to invent missing policy dates or authority levels.
Evidence item
Help article
Product policy
Stated rule
30 days
14 days
Scope
General purchases
A named product category
Effective period
Missing from retrieved passage
Explicit start date in the full document
Authority
Needs confirmation
Approved policy owner identified
Applicability to this purchase
Unresolved
Depends on category and purchase date
The missing fields are part of the answer. They show which question must be resolved before the application can offer a confident eligibility decision.
Use an explicit resolution policySuppose the business owner confirms that the approved product policy takes precedence for purchases in that category after its effective date. Encode that rule where the application can apply and test it. Retrieve the purchase date and category through an authorized path, evaluate the rule, and preserve the evidence used.
If the purchase predates the policy, the system needs the version that applied at that time. If the category is unknown, it should ask for the missing detail or route the case to someone who can establish it. If both sources appear authoritative and no precedence rule exists, report the conflict rather than blending the values into an invented compromise.
An appropriate response might say: “I found a general 30-day policy and a category-specific 14-day policy. I need the product category and purchase date to determine which applies.” This communicates the unresolved condition without pretending that a smooth answer is a verified one.
Keep permissions in the retrieval pathDo not solve a conflict by exposing a source the requester is not allowed to read. Permissions must remain effective when documents are indexed, retrieved, summarized, or cited. OWASP's RAG security guidance discusses access control and the treatment of retrieved material as untrusted input. Primary reference: OWASP RAG Security.
For this example, separate policy text from instructions about what the assistant should do. A sentence embedded in a retrieved document cannot grant permission to issue a refund or reveal another customer's purchase.
Evaluate the decision, not only the proseCreate a small case set: a purchase inside the category, one outside it, one before the effective date, one with missing metadata, and one where a restricted document appears highly relevant. For each case, define the expected applicable source, unresolved condition, permitted output, and allowed action.
For an interview: Explain how retrieval, policy resolution, and response generation have different responsibilities. For a team: Ask the policy owner to review the conflict table and precedence rules. A useful evaluation checks whether the system selected an applicable rule or correctly preserved uncertainty. Citation presence by itself cannot establish that result.
Teaching scenario: The fictional return policies below are created for this exercise. They are not policies of a real business or interpretations of legal requirements.
Relevance does not settle the decisionA customer asks whether an item can be returned after 20 days. Retrieval finds two passages. A general help article says returns are accepted within 30 days. A product-specific policy says the limit is 14 days. Both passages discuss the question, but their presence in the results does not establish which one applies to this purchase.
The system needs more than a plausible sentence and a citation. It needs enough context to determine whether the rule covers the product, transaction, customer, region, and relevant time. A newer document can describe a new policy without governing an older purchase.
Microsoft's RAG overview describes retrieval as grounding responses in external content and highlights the need for authorized, relevant information. That grounding is a starting point; deciding applicability still requires your application's rules and context. Primary reference: Microsoft RAG overview.
Build an evidence recordFor the exercise, capture the passages alongside their metadata. Do not ask the model to invent missing policy dates or authority levels.
Evidence item
Help article
Product policy
Stated rule
30 days
14 days
Scope
General purchases
A named product category
Effective period
Missing from retrieved passage
Explicit start date in the full document
Authority
Needs confirmation
Approved policy owner identified
Applicability to this purchase
Unresolved
Depends on category and purchase date
The missing fields are part of the answer. They show which question must be resolved before the application can offer a confident eligibility decision.
Use an explicit resolution policySuppose the business owner confirms that the approved product policy takes precedence for purchases in that category after its effective date. Encode that rule where the application can apply and test it. Retrieve the purchase date and category through an authorized path, evaluate the rule, and preserve the evidence used.
If the purchase predates the policy, the system needs the version that applied at that time. If the category is unknown, it should ask for the missing detail or route the case to someone who can establish it. If both sources appear authoritative and no precedence rule exists, report the conflict rather than blending the values into an invented compromise.
An appropriate response might say: “I found a general 30-day policy and a category-specific 14-day policy. I need the product category and purchase date to determine which applies.” This communicates the unresolved condition without pretending that a smooth answer is a verified one.
Keep permissions in the retrieval pathDo not solve a conflict by exposing a source the requester is not allowed to read. Permissions must remain effective when documents are indexed, retrieved, summarized, or cited. OWASP's RAG security guidance discusses access control and the treatment of retrieved material as untrusted input. Primary reference: OWASP RAG Security.
For this example, separate policy text from instructions about what the assistant should do. A sentence embedded in a retrieved document cannot grant permission to issue a refund or reveal another customer's purchase.
Evaluate the decision, not only the proseCreate a small case set: a purchase inside the category, one outside it, one before the effective date, one with missing metadata, and one where a restricted document appears highly relevant. For each case, define the expected applicable source, unresolved condition, permitted output, and allowed action.
For an interview: Explain how retrieval, policy resolution, and response generation have different responsibilities. For a team: Ask the policy owner to review the conflict table and precedence rules. A useful evaluation checks whether the system selected an applicable rule or correctly preserved uncertainty. Citation presence by itself cannot establish that result.