What Is RAG? Retrieval-Augmented Generation in Production

Retrieval-augmented generation means searching for relevant documents before answering, and putting them in front of the model. The model then answers from the text you just showed it rather than from whatever it remembers from training. That is the one-sentence version. Behind that sentence sit a dozen engineering decisions that everyone building a production system runs into.

This article walks those decisions in order: how to split documents, which embedding model to pick, why vector search alone is not enough, how many chunks to send, and how to tell whether retrieval or the model is the thing that failed. For the underlying concepts, start with LLM-based application development — everything here builds on it.

Why RAG exists: the model's three blind spots

A language model cannot know three kinds of thing, and all three are why RAG exists.

• Anything after training. Data stops at a date. Yesterday's price change and last month's policy update are invisible.
• Anything specific to you. Internal documentation, customer records, your product catalogue — none of it was in the training set.
• Anything rare. A topic that appears fifty times on the internet is remembered vaguely; one that appears fifty thousand times is remembered sharply. In niche domains recall degrades fast.

What those three have in common is that the model does not leave the gap empty. It fills unknown territory with a probabilistic average of what it does know, and the result is fluent, confident and wrong. RAG puts the right text in the gap so there is nothing to invent.

When not to build RAG

RAG became fashionable, so it gets built where it is not needed. Adding search to every query adds latency, cost and maintenance. Stop and think first in these cases:

• The model already answers correctly. On general-knowledge questions RAG usually adds no quality, only delay.
• There genuinely are few documents. With a ten-page handbook, putting the whole thing in context is simpler, cheaper and more accurate than building search.
• The answer lives in structured data. "This customer's last three orders" is a database question. Write SQL, not a vector query.
• The questions are predictable. If the top ten questions have fixed answers, an FAQ and a decent search box do better.

The right question is not "should we add RAG" but "is there information the model cannot know and that keeps changing". If the answer is no, the layer you are adding is pure overhead.

Two halves of the pipeline

RAG systems are two pipelines running at two different times, and separating them in your head saves hours when something is wrong.

1. 1Indexing (offline): fetch the document → extract text → chunk → embed → write to the vector store. This runs periodically as data changes.
2. 2Query (online): take the question → embed → search for similar chunks → rerank → assemble context → call the model → validate the answer. This runs on every request.

Most quality problems live where nobody looks — in indexing — rather than where everybody looks, which is the query side. If your PDF extraction mangles tables, nothing you do at query time will repair it.

Not every message needs a search: the routing layer

In most production RAG systems the proportion of unnecessary searches is surprisingly high. "Hi", "thanks" and "what did you just say" all trigger retrieval — each one an embedding call, a vector query and a few thousand tokens added to context.

A small routing layer fixes that cheaply. Sort the message into three buckets: needs search (an information question), does not (greeting, thanks, meta-questions about the system) and answerable from history (a follow-up about something already retrieved). A small cheap model is enough for version one; even a rule-based prefilter captures the first slice of the benefit.

ROUTE_SYSTEM = """Classify the user message. Return only this JSON:
{"route": "search" | "chat" | "followup"}

search   : a question seeking new information
chat     : greeting, thanks, meta-question about the system
followup : refers to something in the previous answer ("and the second one?")"""

route = classify(message)          # small, cheap model
if route == "search":
    chunks = retrieve(rewrite_query(message, history))
elif route == "followup":
    chunks = last_turn_chunks          # already in hand, no second search
else:
    chunks = []                        # no retrieval at all

Adding this layer without measuring it is its own risk: a misrouted information question gets answered with no sources, which raises hallucination risk. Put the router in your golden set — a wrong "chat" decision costs far more than a wrong "search" one.

Document preparation: the dullest and most decisive step

Source documents are rarely clean text. PDF, Word, HTML, wiki pages, support tickets, code repositories — each has its own way of degrading.

• Tables. Naive PDF extractors flatten a table row by row and the column relationship disappears. In a pricing document that means a wrong price. Preserve tables as Markdown tables.
• Heading hierarchy. Which section a chunk sits under is half its meaning. Prepend the heading chain: without Installation > Linux > Requirements, the sentence "at least 8 GB" is about nothing in particular.
• Code blocks. Do not blend code into prose; indentation and line breaks carry meaning.
• Boilerplate. Page footers, navigation, cookie banners — they contaminate every chunk and pollute search. Strip them before indexing.
• Date and version. When a document was written and which version it describes belong in metadata. Old documentation mixing with new is the most insidious source of wrong answers.

Chunking: how big, and where to cut

You cannot put a whole corpus in context, so you split documents and retrieve only the relevant pieces. Chunk size is the single parameter that most affects RAG quality.

Two practical rules. First: cut on meaningful boundaries — paragraph, heading, list item — not on a fixed character count. A chunk severed mid-sentence produces two half-ideas. Second: use overlap. Ten to fifteen percent between adjacent chunks stops information that straddles a boundary from vanishing entirely.

def chunk(text: str, target: int = 700, overlap: int = 80) -> list[str]:
    """Chunks near a target size while respecting paragraph boundaries."""
    paras = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, cur = [], []
    size = 0
    for p in paras:
        n = len(p.split())
        # A paragraph larger than target becomes its own chunk rather than
        # being split — splitting is what wrecks tables and code blocks.
        if n > target:
            if cur: chunks.append("\n\n".join(cur)); cur, size = [], 0
            chunks.append(p)
            continue
        if size + n > target and cur:
            chunks.append("\n\n".join(cur))
            tail = cur[-1].split()[-overlap:]        # overlap
            cur, size = [" ".join(tail)], len(tail)
        cur.append(p); size += n
    if cur: chunks.append("\n\n".join(cur))
    return chunks

Enriching the chunk: the parent-child pattern

Small chunks search well, large chunks answer well. A common pattern satisfies both: search the small one, send the large one. You embed and search 200-token chunks; when one matches, you hand the model the wider section it belongs to.

A second variant adds a short context sentence to every chunk at index time: "This section describes the installation steps for product X version Y." A model writes that sentence once and you store it alongside. The cost is one-off at indexing; the benefit repeats on every search, because the chunk now stands on its own.

Choosing an embedding model

An embedding turns text into a vector — typically a few hundred to a few thousand dimensions. Semantically similar texts land near each other in that space, and search is a measurement of that nearness.

• Language coverage. If your content is not in English, pick a multilingual model or one with measured performance in your language. A model trained only on English degrades noticeably elsewhere.
• Dimensionality. Higher means slightly better quality and distinctly more storage and slower search. Mid-size is enough for most workloads.
• Cost and placement. Embedding through an API is easy and bills on every reindex. Self-hostable open models save meaningfully on large collections.
• Asymmetric search support. Questions and documents are different shapes; some models expect distinct prefixes ("query:", "passage:"). Skipping that quietly costs you quality.

The vector store: where things live

You need somewhere to keep vectors and run similarity search. Broadly three options:

Practical advice: start with the database you already run. Up to a few hundred thousand chunks, an extension like pgvector does the job and you avoid operating another service. You migrate when scale genuinely demands it — and by then you will know which features you actually need. Vector database engineering covers that side properly.

How similarity search works

You embed the query with the same model and look for the N nearest vectors. The usual measure is cosine similarity: the cosine of the angle between two vectors, closer to 1 meaning more alike.

Comparing against millions of vectors one by one is expensive, so stores use approximate nearest-neighbour search: index structures such as HNSW trade a little accuracy for a great deal of speed. That trade-off is tunable — change the index parameters and both speed and recall move. Start with defaults, measure, tune if needed.

$ psql -c "explain analyze
    select id, 1 - (embedding <=> :q) as score
    from chunks order by embedding <=> :q limit 5;"
 
 Limit  (cost=0.00..8.12 rows=5)
   ->  Index Scan using chunks_embedding_hnsw on chunks
 Execution Time: 7.214 ms
# ✓ The index is being used.
 
$ # Without it:
 Seq Scan on chunks  (cost=0.00..41233.00 rows=412330)
 Execution Time: 2841.006 ms
# → A 400x difference. Always verify the index exists.

Why vector search alone is not enough

Semantic search is powerful with one weakness: it is bad at exact matches. Product codes, error codes, person names, version numbers — these carry identity, not meaning. The vector for "ERR-4412" is nearly identical to the one for "ERR-4413".

The answer is hybrid search: run classical keyword search (BM25 or similar) alongside vector search and merge. The most robust merge blends the two rankings rather than the two scores, because scores from different systems are not on comparable scales.

def reciprocal_rank_fusion(*ranked_lists, k: int = 60):
    """Merges RANKINGS, not score scales."""
    scores: dict[str, float] = {}
    for lst in ranked_lists:
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

final = reciprocal_rank_fusion(
    vector_search(query, limit=50),
    keyword_search(query, limit=50),
)[:20]

Reranking

Search returned fifty candidates; you cannot send them all. Which five are genuinely relevant? Embedding search is fast but coarse — it encodes query and document separately, so it cannot see the fine-grained relationship between them.

A reranker (cross-encoder) scores query and candidate together and is far more accurate. It is slow, which is why it runs over fifty candidates rather than the whole collection. That two-stage shape — broad and fast, then narrow and accurate — is the standard in modern RAG.

Measure the gain on your own data rather than trusting a published number; how much reranking buys you depends heavily on the corpus. What is consistent is the direction: retrieving more candidates and reranking beats retrieving fewer and trusting the first pass.

Assembling context: how many, in what order

You have five ranked chunks. How you place them in the prompt matters more than you would guess.

• Count: start between three and five. More rarely improves quality and reliably increases cost and latency. Sending ten is an expensive way to distract the model.
• Order: models attend more to the beginning and the end of a context than the middle. Do not bury the most relevant chunk in the middle.
• Separation: give each chunk its own tagged block with its source. Pasting them into one blob loses track of which fact came from where.
• Budget: set a token ceiling for documents and drop the lowest-scoring chunk when it is exceeded — not the conversation history.

def build_context(chunks: list[Chunk], budget: int = 3000) -> str:
    parts, used = [], 0
    for i, c in enumerate(chunks, start=1):
        block = (
            f'<source id="{i}" doc="{c.doc_title}" '
            f'section="{c.heading_path}" updated="{c.updated_at}">\n'
            f"{c.text}\n</source>"
        )
        n = count_tokens(block)
        if used + n > budget:
            break                      # lowest-scoring chunks fall out naturally
        parts.append(block); used += n
    return "\n\n".join(parts)

The prompt side: citations and permission not to know

You retrieved the right documents. You are not finished. Two instructions are worth more than the rest.

First, require citations. Ask the model to mark each claim with a source number. That does three things: users can verify, you can debug, and — most usefully — a model that knows it must cite invents less.

Second, permission not to know. If the retrieved documents do not contain the answer, the model must be able to say so. Without explicit permission it will manufacture something out of whatever it has. One sentence measurably cuts hallucination.

SYSTEM = """You are a documentation assistant.

Rules:
1. Use ONLY the content of <source> blocks. Do not use general knowledge.
2. Mark each claim with its source number: [1], [2].
3. If the sources do not answer: "I could not find that in our records."
   Do not guess or infer from adjacent topics.
4. If sources conflict, say so and prefer the one with the newer date.
"""

Metadata filtering and authorisation

This is the most-skipped and most dangerous part of RAG. Vector search returns semantically nearest chunks — it has no opinion about whether this user may read them. Index HR documents and salary data in the same collection and anyone asking the right question can see them.

# WRONG: search, then filter
hits = vector_search(q, limit=5)
visible = [h for h in hits if can_read(user, h.doc_id)]
# → If 4 of 5 get filtered you are left with one chunk, quality collapses,
#   and you have already leaked which documents exist.

# RIGHT: push the filter into the search
hits = vector_search(
    q,
    limit=5,
    where={"acl_group": {"$in": groups_of(user)}},   # store-level filter
)

Filtering at the store level gets you the right number of results and guarantees unauthorised content never enters the process at any stage. In a multi-tenant system put the tenant id in the same filter — and in the cache key, or one tenant's answer will be served to another.

Multi-tenant setups

Serving several customers raises one question: a single collection, or one per tenant?

• One collection plus a tenant filter. Simple to operate, cheap. The risk is the filter being forgotten on one code path, and that leak is silent. Enforce it in the store access layer so callers cannot omit it.
• One collection per tenant. Strong isolation, easy deletion requests (drop the collection). Operationally heavy past a few hundred tenants.
• Hybrid. Small tenants share, large or regulated tenants get their own. Most SaaS products end up here.

Whichever you choose, the tenant id belongs in three places: the search filter, the cache key and the log line. The third is for debugging; the first two are what prevent a leak.

Evaluation: measure retrieval and generation separately

The commonest mistake in RAG evaluation is measuring the system as one box. A bad answer gets blamed on the model and somebody starts editing the prompt — when in most cases the right document simply never arrived.

Measuring retrieval needs a hand-built set: question plus the id of the document that answers it. Fifty questions is a sufficient start. That set is the only thing that will tell you whether changing chunk size or embedding model helped. For the wider evaluation setup see the golden-set section in LLM-based application development, and AI agent evaluation and observability for depth.

The order in which to diagnose a bad answer

When a user reports a wrong answer, there is an order to where you look. Skipping it and going straight to the prompt is the single biggest time sink in RAG projects.

1. 1Is the right document in the collection at all? If not, the problem is indexing and nothing downstream will help.
2. 2Did search return it? If not, the problem is retrieval: chunking, embedding, hybrid search, reranking.
3. 3Did it return but rank low? In the top fifty but not the top five is exactly where a reranker earns its place.
4. 4Did it make it into context? It may have been cut by the token budget. Log the budget and the trimming decisions.
5. 5Did the model use what it was given? If everything above is right and the answer is still wrong, now it is a prompt problem.

Running those five steps quickly requires logging retrieval detail on every request: the query issued, which chunk ids came back with which scores, which entered context. Without that log, diagnosis becomes guesswork. A debug endpoint — restricted to authorised users — turns it into seconds.

$ curl -s localhost:8080/debug/retrieve \
    -d '{"q":"certificate renewal period"}' | jq -r '.hits[] | "\(.score)  \(.doc)  \(.heading)"'
0.812  cert-policy-v3.md  Renewal > Periods
0.774  cert-policy-v2.md  Renewal > Periods
0.603  onboarding.md      FAQ
# → v2 and v3 both came back: the old version is still indexed.
# ✓ The diagnosis is in retrieval, not the prompt: no version filter.

A case: search quietly returning nothing

An internal documentation assistant has run well for months. One day users start saying it does not know anything any more. The model answers every question with "I could not find that in our records" — which is to say, it is following its instructions correctly.

$ curl -s localhost:8080/debug/retrieve -d '{"q":"VPN setup"}' | jq '.hits | length'
0
 
$ psql -c "select count(*) from chunks;"
 412330
# → The chunks are there.
 
$ psql -c "select count(*) from chunks where embedding is null;"
 412330
# → But none of them has a vector.
 
$ kubectl logs job/reindex --tail=5
  embedding provider returned 429 (rate limited), retrying...
  embedding provider returned 429 (rate limited), retrying...
  giving up after 5 attempts; wrote 412330 rows with null embedding
# ✓ The reindex job hit a rate limit — and exited zero anyway.

The root cause: the nightly reindex job hit the embedding provider's rate limit. The error was caught, logged and — the critical failure — the job reported success. Old vectors had been deleted and new ones never written. The system was technically working: search returned nothing and the model correctly said it did not know.

The fix had three parts. The job must not exit zero on partial success. The proportion of rows with null embeddings becomes a monitored metric. And most importantly, the new index is written without deleting the old one and swapped only after validation passes. Blue-green deployment, applied to data.

Freshness: what happens when data changes

Documents change, get deleted, get added. The index has to keep up, and there are three common patterns.

• Full reindex. Simple and expensive. Fine as a nightly job on small collections, unsustainable as they grow.
• Incremental update. Reprocess only what changed. The best option when the source system exposes modification times or webhooks. Do not forget to remove deleted documents — that is the most commonly skipped step.
• Versioned index. Write the new index to a separate collection, validate, then swap. The remedy from the case above; it should be standard for large reindexes.

The second freshness question is what you tell the user. Showing which document and which date an answer came from raises trust considerably. "From the installation guide dated 14 March 2026" is far more useful than the same answer with no date.

Tables, numbers and mixed content

Text-based RAG works well on prose. Table-heavy and number-heavy documents need a different strategy: embedding a pricing row as free text is nearly useless for search, because the vector for "399" carries no meaning.

• Turn the table into sentences. At index time generate one sentence per row: "The starter plan is 399 a month and includes 5 users and 10 GB of storage." That sentence is what you search; you hand the model both it and the original row.
• Keep small tables whole. Do not chunk them; make the whole table one chunk and preserve the header row. A split table produces wrong answers because the column relationship is gone.
• Route numeric questions away. "How many", "total", "highest" belong in a structured query, not a vector search. Accepting that RAG is structurally weak here beats forcing it.

Scanned documents and visual content

A meaningful share of corporate archives is scanned PDF: no text layer, the page is an image. You cannot index those without OCR, and OCR quality converts directly into RAG quality.

Two things matter. First, OCR errors are silent: "1000" read as "l000" breaks both search and answer with nothing visibly wrong. Store the OCR confidence at index time and flag low-confidence pages. Second, on pages carrying diagrams, text alone is insufficient; generating a description of the image with a model and appending it to the text makes those pages markedly more findable. Multimodal AI covers that ground.

Cost and latency

RAG generates cost in two places that behave differently. Indexing cost scales with collection size: infrequent and predictable. Query cost scales with traffic and repeats on every request — an embedding call, a search, and the tokens of whatever you added to context.

• Cache answers for frequent questions — same question, same answer, free and instant. Remember to key the cache by tenant and permission group.
• Experimentally reduce chunk count; going from five to three with no quality loss removes forty percent of your context cost.
• Cache embeddings so a repeated query does not recompute its vector.
• Stream — it transforms perceived latency.

The limits of RAG

RAG does not solve everything, and knowing its edges keeps you from investing in the wrong place.

• It is weak at aggregation and counting. "How many customers do we have" has no answer written in any document, so similarity search cannot find one. Those are database questions.
• One shot is not enough for multi-step reasoning. "How does feature B of product A behave in version C" may need several rounds of retrieval. That is where the work shifts toward agents — see how AI agents work.
• It cannot exceed your documentation. If the docs are incomplete, RAG makes that visible; it does not fill the gap. The real output of many RAG projects is a measurement of how bad the documentation is.
• It does not fix tone or format. Changing how the model writes is prompt work or fine-tuning, not a retrieval concern.

Choosing sources: not indexing everything

The first instinct is to index all corporate content. That almost always lowers quality. Abandoned drafts, notes from cancelled projects, meeting minutes from three years ago — all of it competes with real documentation in search results, and sometimes wins.

Make indexing a deliberate decision. Three questions per source: is it maintained, does it have an owner, is it trusted. Anything that fails one of those stays out, or goes in a separate collection weighted down at search time. Breadth looks attractive; in practice a narrow clean collection beats a broad dirty one by a clear margin.

The same logic applies to deletion. When a document is removed at source it must leave the index; that is the most commonly skipped step in incremental updates, and the result is an assistant explaining a policy that no longer exists.

Advanced variants, and when they are warranted

Once the basics are built and measured, there are directions to explore where you are genuinely stuck. Order matters: attempted before the basics work, these only add complexity.

Query rewriting

The user's question may be poorly formed for search: too short, context-dependent ("and the second one?"), or several questions at once. Having the model rewrite or decompose the query before searching makes a visible difference in multi-turn conversations.

Multi-round retrieval (agentic RAG)

The model searches, reads the result, notices what is missing and searches again. It improves quality on complex questions and increases both latency and cost. Always cap the number of rounds.

Graph-assisted retrieval

Model relationships between documents as a graph and expand search along it. Valuable where relationships between entities are the point — org charts, component dependencies. For general documentation search the payoff rarely covers the setup.

Team and process

The non-technical failure mode of RAG projects is always the same: nobody owns the documentation. The system launches, works well for a week, then the documents go stale and nobody updates them. Six months later the assistant is explaining an old policy and nobody knows when it started being wrong.

Three practices prevent that. Show the source document and its date on every answer, so a stale document becomes visible to users. Report the twenty most-retrieved documents periodically — those are the spine of the system and the first ones to keep current. And log every "I could not find that" answer: that list is a free content roadmap telling you exactly where the documentation has holes.

Common questions

Context windows grew — is RAG still necessary?

Maybe not, if your corpus is small; a hundred-page handbook now fits. Three constraints remain: you pay for that context on every request, latency grows with it, and the tendency to miss material in the middle does not fully disappear. With tens of thousands of documents you still need retrieval.

Should I put a threshold on the retrieval score?

Yes, carefully. A threshold keeps irrelevant chunks out of context and makes abstention easier. The risk is that set too high it also drops correct-but-low-scoring documents. Calibrate it against your retrieval set — measure how many correct answers you lose while reducing wrong ones. A relative threshold based on the score distribution usually survives better than a fixed number.

What chunk size is best?

There is no universal answer and most of it depends on content type. 500-800 tokens is a reasonable start; tune from there against your fifty-question set. This parameter is learned from your data, not from a best-practices list.

Do I really need query rewriting?

Rarely for single-turn search. Almost always for multi-turn: the vector for "and how much is it" is useless because it does not know what "it" is. Rewriting into a standalone question using the conversation history fixes that. Small model call, clear payoff.

I have documents in several languages — what now?

Use a multilingual embedding model so a question in one language can find a document in another. Alternatively keep a collection per language, detect the query language and route — more work, but it lets you use the best model per language.

Can I search several collections at once?

Yes, and most mature systems do: documentation, support tickets and the product catalogue sit in separate collections, the query hits all of them and results merge. The advantage is per-collection chunking and freshness policies. Merge by blending rankings again; scores across collections are not directly comparable.

How long does building RAG take?

A working first version, a few days. A production-grade system — authorisation, freshness, evaluation, monitoring — weeks. Most of the time goes not into the model but into document preparation and evaluation, and both of those take about twice as long as anyone estimates.

A pre-launch checklist

1. 1I have read raw extraction output by hand; tables and headings survive.
2. 2Chunks carry heading chain and date metadata.
3. 3Hybrid search is on; I am not relying on vector search alone.
4. 4The authorisation filter is inside the search query, not applied afterwards.
5. 5I have a fifty-question retrieval set and a measured recall@5.
6. 6The model must cite sources and is able to say it does not know.
7. 7Zero-result query rate and null-embedding rate are monitored.
8. 8Reindexing fails loudly on partial success; a new index is validated before the swap.

If any of those eight is not yes, the system is not ready — working in a demo does not change that. Item four especially: authorisation is not something you add later, it is part of search from the beginning.

The smallest useful starting point

All these lists describe a comprehensive system. Your first version does not have to be one. A weekend's worth of working, useful RAG needs: a folder of Markdown documents, paragraph-boundary chunking, an embedding model, PostgreSQL with pgvector, and a fifteen-line search function.

The most valuable thing that first version gives you is not code but measurement: what recall@5 looks like on your data, which questions go unanswered, how messy the documents really are. Architecture debates held before those three facts are known rest on guesswork. Measure first, complicate second — every advanced technique in this article is worth something only when it answers a measured deficiency.

Next step

RAG closes the model's knowledge gap. But reading and answering is sometimes not enough: when a user wants something done, the model has to move beyond producing text. Tool use, decision-making and multi-step tasks are their own subject: how AI agents work.

One last reminder: none of the techniques here produces good RAG on its own. Quality is capped by the weakest link in a chain running from document preparation to authorisation. Build the chain end to end, then start measuring, then improve one thing at a time — rerunning the same fifty questions each time. That loop is the shortest path to learning this properly.