# RAG Evaluation: Test the Questions Your Business Asks > Evaluate RAG using realistic business questions, source evidence, access boundaries and useful abstention rather than a few polished demonstrations. - URL: https://weaveward.com/blog/rag-evaluation-business-questions - Published: 2025-11-06 - Updated: 2026-10-12 - Category: AI - Tags: RAG evaluation, knowledge assistants, AI testing - Author: Weaveward Team A retrieval-augmented generation system retrieves documents and uses them to produce an answer. Testing a few questions chosen by its developer can make it look dependable while leaving ordinary business failures undiscovered. An evaluation set should represent the questions users ask, the evidence available to answer them and the conditions in which the assistant should decline. Separate retrieval quality from answer quality. A correct document can be retrieved but interpreted incorrectly. A fluent answer can also be produced when the relevant document was never retrieved. Those failures require different fixes. ## Build a question set from real work Collect approved, anonymized questions from support requests, internal searches and interviews. Include short questions, ambiguous wording and references to product or policy versions. Do not simply rewrite document headings into questions; employees rarely use the same wording as the knowledge base. For each question, identify the expected source, important facts and acceptable behavior when the source is unavailable. Ask a subject-matter owner to review the reference answer. Record which user role is allowed to see the evidence, because a factually correct answer can still be an access-control failure. ## Worked example: two return policies Suppose a hypothetical retailer has an old return policy and a newer policy effective for a particular product range. A customer-service employee asks whether an opened item can be returned. The question omits the product and purchase date. A useful assistant identifies the missing context and asks for it, or explains the relevant uncertainty with citations. An answer copied from the older policy fails even if every sentence is supported by that document. The evaluation needs to check policy applicability, not only whether the model quoted a retrieved passage faithfully. Add a related question from a user who lacks access to an internal exception policy. The assistant must not reveal that restricted policy through its answer, citation title or retrieved snippets. Test both the retrieval layer and the final response under the intended identity. ## Score distinct dimensions [LangSmith's RAG evaluation tutorial](https://docs.langchain.com/langsmith/evaluate-rag-tutorial) separates dimensions including correctness, groundedness and retrieval relevance. [Microsoft's generative AI evaluation guidance](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-approach-gen-ai) similarly treats evaluation as a set of measures rather than one universal score. For a business pilot, track whether the necessary source was retrieved, whether the answer uses that source accurately and whether the user can act on the response. Add an abstention measure for questions outside the available knowledge. A refusal is desirable when it prevents an unsupported instruction, but excessive refusals can make the tool unhelpful. ## Include failures the demonstration avoids Test contradictory documents, expired procedures, scanned files with extraction errors and questions that require several sources. Include documents containing instructions aimed at the model, so the system treats retrieved text as evidence rather than permission to change its behavior. Use representative document access settings and a realistic update process. If a removed document remains searchable for hours, record that exposure rather than evaluate only the clean newly indexed dataset. Refresh failures and stale versions are part of the product's behavior. ## Compare changes without changing the test Keep a reviewed holdout set separate from the questions used to tune prompts or retrieval. Record the corpus version, model, retrieval settings and evaluation method for each run. Automated graders can help sort failures, but review significant cases with knowledgeable people. Set release criteria according to the workflow's consequences and provide a route to a human. A useful RAG evaluation makes the system's evidence, uncertainty and access boundaries visible. It gives the team a reason to trust a particular deployment for a particular task, without implying reliability for every question. ## Primary references - [LangSmith: Evaluate a RAG application](https://docs.langchain.com/langsmith/evaluate-rag-tutorial) - [Microsoft: Generative AI evaluation approach](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-approach-gen-ai) ## Related reading - [What is RAG for business documents?](/blog/what-is-rag) - [Evaluate AI accuracy before deployment](/blog/evaluate-ai-accuracy-before-deployment) Develop a practical evaluation plan with our [AI solutions service](/services/ai-solutions).