A retrieval-augmented generation prototype can look convincing within a day. Production is a different problem.
Evaluate before you scale
Create a test set of real questions with expected sources and answers. Measure retrieval quality (did the right passages come back?) separately from generation quality (did the model use them faithfully?). Track both across every change.
Respect access control
Retrieval must apply the same permissions as the source systems. If a user cannot open a document, the assistant must not summarize it. Enforce this at query time, not after generation.
Keep data fresh
Define how documents are ingested, chunked, re-embedded and deleted. Stale or duplicated content is a frequent root cause of wrong answers.
Design for failure
Show sources, allow users to flag bad answers, and have the system say when it does not know. Add rate limits, timeouts and fallbacks for provider outages.
Watch cost and latency
Token usage grows with context size. Use caching, smaller models for simple steps and limits on retrieved context to keep both predictable.