Technology Insights

Retrieval-Augmented Generation in Production: What Changes After the Demo

Evaluation, data freshness, access control and cost considerations when taking RAG applications live.

April 2, 20268 min readBy Northbeam Engineering

A retrieval-augmented generation prototype can look convincing within a day. Production is a different problem.

Evaluate before you scale

Create a test set of real questions with expected sources and answers. Measure retrieval quality (did the right passages come back?) separately from generation quality (did the model use them faithfully?). Track both across every change.

Respect access control

Retrieval must apply the same permissions as the source systems. If a user cannot open a document, the assistant must not summarize it. Enforce this at query time, not after generation.

Keep data fresh

Define how documents are ingested, chunked, re-embedded and deleted. Stale or duplicated content is a frequent root cause of wrong answers.

Design for failure

Show sources, allow users to flag bad answers, and have the system say when it does not know. Add rate limits, timeouts and fallbacks for provider outages.

Watch cost and latency

Token usage grows with context size. Use caching, smaller models for simple steps and limits on retrieved context to keep both predictable.

Let’s talk about what you want to build

Share your goals and constraints. An engineering lead will respond with next steps and a suggested engagement model.

Contact us