Almost every enterprise has tried RAG. Most have a notebook somewhere where it worked on a curated set of documents and a hand-picked question list. The painful gap is between that notebook and the system a thousand colleagues actually use against a moving document corpus, three months later, when the team that built it has moved on. This post is the architecture that closes that gap, expressed in OCI terms — but the patterns transfer to any cloud.
The five things that make production RAG hard
It is worth naming the friction explicitly, because the differences between a demo and a production system are not exotic:
- Corpus drift. Documents change, get added, get deleted. The index must follow without manual intervention.
- Retrieval quality at scale. Pure vector similarity is not enough once the corpus is large. You need hybrid retrieval, reranking, and signal from metadata.
- Eval that survives release after release. "It looks right when I tested" is not eval. You need a held-out question set, ground-truth answers and automatic scoring.
- Guardrails. Refusing to answer when the corpus does not support a claim is more valuable than a confident wrong answer.
- Cost. Without explicit ceilings, RAG systems get expensive in surprising ways. Inference, embeddings, vector storage, retraining — all line items that grow.
An architecture that does not have a story for each of these is a demo, regardless of how impressive it looks.
The reference architecture on OCI
A pattern that has held up across several DACH enterprise deployments:
Source layer. Documents live in OCI Object Storage, partitioned by sensitivity and source system. SharePoint, Confluence, file shares and database exports all flow into curated buckets — not directly into the index.
Ingestion pipeline. A scheduled job (OCI Functions or a small Kubernetes job) detects changes in the source buckets, runs them through a normalisation step (text extraction, OCR for scanned PDFs, metadata enrichment), and writes structured chunks to a staging area.
Embedding and indexing. Chunks are embedded with a chosen embedding model (often the same OCI Generative AI embedding endpoint your application calls; sometimes a self-hosted model for sensitive data). Embeddings land in a vector index — Oracle AI Vector Search inside the database is the natural choice for OCI-anchored teams, with the operational benefits of keeping vectors next to the data.
Retrieval layer. A retrieval service exposes hybrid search: vector similarity combined with keyword (BM25-style) relevance, filtered by metadata (sensitivity, document type, last-updated). A reranker then orders the top results before they reach the LLM.
Generation. The retrieved context, the user's query and a system prompt go to OCI Generative AI (or a self-hosted model for the most sensitive use cases). The response is post-processed to extract citations and confidence signals.
Eval and observability. Every query, retrieved context, generated answer and user feedback signal flows into a structured log. A daily eval job runs the held-out question set and emits quality metrics.
That is six components. Each is replaceable. None of them is exotic. The architecture's strength is in the boundaries, not the individual pieces.
What "hybrid retrieval" really means and why it matters
Pure vector similarity wins for semantic similarity ("find me documents about contract termination" returns documents that talk about ending agreements even if they do not use the exact word). It loses for exact-match queries ("find me contract 2024-718") and for queries where a specific term carries weight.
Hybrid retrieval combines vector with keyword (BM25 or similar) and weights the two by query characteristics. The hard work is not in implementing either approach; it is in tuning the weighting and in deciding which metadata filters to apply. The pattern that holds up: start with a 70/30 vector-to-keyword weighting, measure on your eval set, tune from there.
The eval discipline that prevents regression
The single difference between a RAG system that improves over time and one that decays is the eval pipeline. A workable minimum:
- A held-out question set of 100 to 300 representative queries with known good answers.
- An automatic scorer — either a smaller cheaper model that judges semantic equivalence, or rule-based scoring for queries where exact matches matter.
- A daily run that emits aggregate quality metrics: hit rate, faithfulness (does the answer match the retrieved context), citation accuracy, refusal rate.
- A regression alarm when any metric drops more than a chosen threshold versus the rolling baseline.
The eval set is curated, not generated. Spend the week to build it once; reap the benefit every release.
Without eval, every release is a guess. With eval, you can tell your CIO "answer quality is up 3% this month" and mean it.
Cost discipline that does not strangle the project
A common mistake: launching without explicit cost ceilings and discovering, mid-quarter, that the bill grew faster than the value. The defensive patterns:
- Embedding caching. Documents change; most of their chunks do not. Re-embed only the chunks that changed.
- Tiered models. Use a cheaper model for retrieval reranking and for the eval scorer; reserve the expensive model for the user-facing answer.
- Context trimming. The agent rarely needs the full top-10 chunks; sending top-3 reranked chunks usually produces the same answer for a quarter of the token cost.
- Per-user soft caps. Most users ask a handful of questions; the long tail of "let me try a thousand prompts" is where unexpected spend hides.
None of these compromise quality. All of them keep the bill explainable.
Guardrails that earn user trust
The most underrated production feature: refusing to answer when the corpus does not support a claim. Users learn fast which assistants confidently hallucinate; once they know yours does, retention dies.
The patterns that work:
- Faithfulness scoring on every response. If the answer is not grounded in the retrieved context above a threshold, refuse and explain.
- Cited passages by default. Every claim shows the document and paragraph it came from. Users self-verify and trust faster.
- Confidence honesty. When retrieval returned weak results, say so. "I could not find a clear answer in our knowledge base for this question" is a feature.
What this looks like at month three
A team that follows this pattern has, by month three of a production deployment:
- An ingestion pipeline running daily with monitored failure rates.
- A vector index covering the agreed corpus, refreshed automatically.
- A hybrid retrieval service with tuned weights.
- A daily eval run with a 100+ question held-out set.
- An observability stack capturing every query and answer.
- A cost dashboard with explicit per-component breakdowns.
That sounds like a lot. It is roughly a six-week build for an experienced team. Compared to a year of "the RAG is acting weird in production again," it is a bargain.
The longer pattern
The reason enterprise RAG is hard is not the AI; it is the same reason any production data system is hard. Pipelines drift, quality degrades, costs grow. The patterns that work are the patterns we already know from data engineering: separation of concerns, explicit boundaries, monitored quality, cost discipline. Apply them and RAG stops being an interesting demo and starts being infrastructure.