Direct answer
Production RAG is a retrieval system with a generated answer layer. It should identify the exact evidence behind an answer, refuse when evidence is insufficient, and be tested against a maintained set of real questions.
Treat RAG as a retrieval system first
The generator can only use what retrieval finds, so corpus quality, query handling and ranking deserve more attention than answer tone.
A polished response can conceal weak retrieval. The model may write a confident synthesis from an outdated policy, a loosely related chunk or no supporting passage at all. That is why the first production question is not “Which model sounds best?” It is “Can the system consistently surface the evidence a careful person would need?”
Separate the pipeline into inspectable stages: ingest, parse, segment, enrich with metadata, retrieve, rerank, assemble context, generate, cite and evaluate. Log the inputs and outputs of each stage. When an answer fails, the team should be able to tell whether the document was absent, parsing lost the relevant text, retrieval missed it, ranking buried it or generation ignored it.
This decomposition also prevents model changes from becoming a universal remedy. A stronger generator cannot recover a policy that never entered the index, and a longer context window does not fix contradictory documents. Repair the stage that failed.
Define a corpus contract before indexing
Decide which sources are authoritative, who owns them, how versions are handled and what should happen when documents conflict or expire.
“Use our documents” is not a corpus specification. List the repositories, file types and collections that are in scope. Mark sources that are authoritative for particular subjects and sources that are only contextual. Exclude drafts, duplicates and material the intended audience is not allowed to access.
Every indexed unit should carry provenance: source title, canonical location, document owner, effective or updated date where available, access scope and a stable identifier. Those fields support filtering, citations, re-indexing and deletion. They also make an answer reviewable after the underlying corpus changes.
Conflicts need an explicit policy. The system may prefer the newest approved version, prefer a designated source over an informal note, or stop and show both sources. Quietly choosing whichever passage ranks first turns a content governance problem into an invisible model decision.
Authority
Which source governs this topic, and can an operator identify its owner?
Freshness
How does an update invalidate or replace older indexed material?
Access
Can retrieval enforce the same audience boundaries as the source system?
Deletion
Can a source and its derived chunks be located and removed without rebuilding an opaque corpus?
Segment for meaning, then preserve context
Chunking should follow the structure of each document type and retain the headings, identifiers and neighboring context needed to interpret a passage.
A fixed character window is a useful experiment, not a universal production rule. A policy manual, product catalogue, support thread and table encode meaning differently. Segment at natural boundaries where possible: sections, clauses, records, question-and-answer pairs or table rows with their headers.
Chunks still need enough surrounding context to stand alone. Preserve the document title and heading path. Carry table headers into retrieved rows. Keep clause numbers and page references when they matter to the reader. If an answer cites a fragment that cannot be understood when opened, the citation is decorative rather than useful.
Test segmentation with real questions. For each question, inspect whether the evidence exists in a retrievable unit and whether that unit contains distracting or contradictory material. Tune chunking per source class instead of chasing a single magic size for the whole corpus.
Make citations part of the answer contract
A citation should identify the exact source passage used for a claim and let the reader verify it with minimal effort.
Adding links after generation is not enough. The generation step should receive stable source identifiers with the context, attach claims to those identifiers, and decline to cite material that was not supplied. The interface should render a useful source label and open the relevant document or passage when permissions allow.
Citation quality has several layers. The source must exist. The referenced passage must support the nearby claim. The source must be appropriate for that claim. And the user must be allowed to see it. A technically valid link to a weak or inaccessible source does not make the answer trustworthy.
Require the system to distinguish supported facts from synthesis. If it combines several passages into a recommendation, label that recommendation as a synthesis and show the evidence. If retrieved sources disagree, surface the disagreement. If the evidence does not answer the question, say so directly instead of filling the gap from model memory.
Build evals from questions the system must answer
Use a versioned set of representative questions, expected evidence and acceptable answer behavior to test retrieval and generation separately.
Start with real information needs from the intended users. Include straightforward lookups, multi-source questions, ambiguous wording, outdated terminology, access-restricted content, unanswerable questions and cases where sources conflict. Remove or protect sensitive material according to the same rules as the production system.
For each case, record the expected source or sources, the key facts an acceptable answer should contain, statements it must not make and whether refusal is correct. Avoid one brittle reference paragraph when several phrasings would be valid. The goal is to evaluate behavior, not reward imitation.
Measure stages independently. Retrieval evaluation asks whether relevant evidence appears in the candidate set and where it ranks. Citation evaluation asks whether each cited passage supports its claim. Answer evaluation asks whether the response is grounded, complete enough for the task and appropriately refuses unsupported requests. Human review remains important for consequential or domain-specific judgments.
Retrieval coverage
Did the correct evidence enter the retrieved and reranked context for the question?
Citation correctness
Does each citation support the claim attached to it, rather than merely sharing keywords?
Grounded answer behavior
Does the answer stay within the supplied evidence, represent uncertainty and avoid unsupported additions?
Refusal quality
When evidence is absent or permissions block it, does the response stop clearly and suggest a safe next step?
Operate retrieval as changing infrastructure
Production readiness requires monitored ingestion, versioned indexes, traceable queries, regression tests and a controlled rollback path.
Documents change, parsers fail and user language shifts. Monitor ingestion for missing files, parsing errors, unusual chunk counts and stale sources. Record the corpus and pipeline version used for every answer so a reported problem can be reproduced.
Run the evaluation set when retrieval logic, embedding models, rerankers, prompts, document parsers or source collections change. Review failures by stage instead of collapsing them into one score. A change can improve average answer style while making a critical policy question harder to retrieve.
Finally, make feedback actionable. A thumbs-down without the query trace and cited evidence tells the team very little. A report tied to the answer, source versions, retrieved candidates and user explanation can become a regression case. That loop—observe, diagnose, add a case, repair and rerun—is what turns a RAG demo into maintained information infrastructure.