A retrieval-augmented generation (RAG) prototype can be built in an afternoon: split some documents, embed them, store them in a vector index and pass the top results to a language model. Getting the same system to answer reliably for thousands of employees, on documents that change daily, with the right people seeing the right content, is a different job.

This checklist covers the engineering that separates a convincing demo from a dependable production system.

1. Start from the questions, not the documents

Most RAG projects begin by indexing everything available. Better results come from starting with the questions users actually ask.

  • Collect 50–200 real questions from the intended users before building
  • Record the correct answer and the source document for each — this becomes your evaluation set
  • Classify questions by type: factual lookup, comparison, procedural, summarisation, "not in the documents"
  • Decide what the system should do when the answer is not in the sources (it should say so)

2. Ingestion and chunking

Retrieval can only find what ingestion preserved.

  • Parse documents with their structure intact — headings, tables and lists carry meaning
  • Chunk along document structure (sections, paragraphs) rather than fixed character counts where possible
  • Attach metadata to every chunk: source, title, section, date, owner, access permissions
  • Handle tables and scanned documents explicitly; they are where naive pipelines fail most
  • Make ingestion incremental, so updated and deleted documents are reflected in the index quickly

3. Retrieval quality

In most failing RAG systems, the model is not the problem — the retrieved context is.

  • Measure retrieval separately from generation: is the right chunk in the top results?
  • Combine semantic (vector) search with keyword search (hybrid retrieval) for names, codes and acronyms
  • Add a re-ranking step to improve precision in the final context
  • Use metadata filters (product, region, date, document type) where users' questions imply them
  • Consider query rewriting for conversational follow-ups ("what about last year?")
  • For questions that need connected facts across documents, consider a knowledge graph alongside vector search

4. Permissions and data security

An assistant that reveals a document to someone who could not open it is a security incident.

  • Enforce the source system's access controls at retrieval time, per user — not just at ingestion
  • Keep permissions in sync when access changes in the source systems
  • Exclude or mask sensitive fields that should never reach the model
  • Log which documents were retrieved for which user, for audit
  • Confirm where prompts, context and outputs are processed and stored, and that this meets data-residency requirements

5. Generation and grounding

  • Instruct the model to answer only from the provided context and to say when the answer is not there
  • Show citations that link to the exact source passage, so users can verify
  • Keep prompts versioned and tested like code
  • Choose the model per use case, balancing quality, latency, cost and data-handling requirements

6. Evaluation: make quality measurable

Without evaluation, every change is a guess. Build an evaluation loop before you tune anything.

What to measure Question it answers How
Retrieval recall Did we retrieve the passage containing the answer? Compare retrieved chunks with the labelled source
Faithfulness Is every claim in the answer supported by the context? LLM-as-judge plus human review of a sample
Answer correctness Does the answer match the reference answer? Reference comparison on the evaluation set
Refusal behaviour Does it decline when the answer is not in the sources? Include unanswerable questions in the set
Latency and cost Is it fast and affordable enough at expected volume? Measure per request, at realistic load
  • Run the evaluation set automatically on every change to prompts, models, chunking or retrieval
  • Calibrate automated judges against human ratings before trusting them
  • Grow the evaluation set continuously from real user questions and feedback

7. Guardrails

  • Detect and handle prompt-injection attempts, including instructions hidden inside retrieved documents
  • Filter inputs and outputs for sensitive data and policy violations
  • Limit the scope: an HR assistant should decline questions about unrelated topics
  • For agentic RAG that can take actions, require human approval for consequential steps

8. Observability and operations

  • Trace every request end to end: query, retrieved chunks, prompt, model, response, latency, tokens
  • Capture user feedback (helpful / not helpful, with a reason) and route it into evaluation
  • Monitor quality, latency, error rates and cost over time, with alerts
  • Version indexes, prompts and models so you can roll back
  • Assign an owner for content freshness — stale documents produce confidently wrong answers

9. Cost control

  • Cache embeddings and, where appropriate, responses to repeated questions
  • Right-size the context: more chunks are not always better and always cost more
  • Route simple questions to smaller, cheaper models
  • Track cost per question and per user group

10. Adoption

The best-engineered assistant fails if people do not trust it.

  • Launch with a focused use case and a defined group of users
  • Explain what the assistant can and cannot answer
  • Make citations prominent, so trust is earned answer by answer
  • Review feedback weekly in the first months and act on it visibly

The short version

Production RAG is a data and engineering problem as much as an AI problem. Get ingestion, retrieval, permissions and evaluation right, and model choice becomes a tuning decision rather than a gamble.

Moving a GenAI pilot into production? Talk to our AI engineering team.