Reviewed 18 Sept 20265 min

RAG: the basics

The previous three posts built the retrieval half of the pipeline: chunk, embed, index, search. This one covers the other half — what retrieval-augmented generation actually does with the chunks it gets back, why grounding an LLM in retrieved text beats relying on its parametric memory, and the failure modes that are unique to this step.

RAG: the basics

A five-minute read.

The vector databases post named retrieval-augmented generation as one of the main reasons any of this exists, then moved on without explaining it. The embedding models and chunking posts spent their time on how to get good chunks back for a query. This post covers what happens after that: handing those chunks to a language model and getting an answer grounded in them.

The idea in one sentence

Instead of asking a language model a question and trusting whatever it remembers from training, retrieve the passages most likely to contain the answer, put them in the prompt, and ask the model to answer from those passages specifically.

flowchart LR
    Q["user question"] --> E["embed + search<br/>(the previous three posts)"]
    E --> C["top-k chunks"]
    C --> P["prompt: question + chunks"]
    P --> L["LLM"]
    L --> A["answer, grounded in the chunks"]

Everything left of “prompt” is the pipeline already covered. RAG is the two boxes after it: build a prompt out of what you retrieved, and let the model’s job shrink from “recall the fact” to “read these passages and answer.”

Why bother, when the model already knows things

A language model’s knowledge is frozen at training time and baked into its weights. That’s a bad fit for anything that’s private, that changes after the model was trained, or that the model needs to point back to a source for. Retrieval fixes all three at once:

Freshness. Re-index a changed document and the next query sees the change. Nothing about the model itself has to move.
Access to private data. The model was never trained on your internal wiki. Retrieval is how it sees it, without fine-tuning anything.
Traceability. A grounded answer can point at doc_id and chunk_index — the same metadata the chunking post described — so a user can go check the source instead of taking the model’s word for it.

The tradeoff is that the answer is now only as good as the passages it was given. That’s the property the rest of this post is about.

Building the prompt

The mechanical part is ordinary string assembly: format each retrieved chunk with enough identifying metadata to cite, and instruct the model to stick to what’s there.

def build_prompt(question: str, chunks: list[dict]) -> str:
    context = "\n\n".join(
        f"[{c['doc_id']}#{c['chunk_index']}]\n{c['text']}"
        for c in chunks
    )
    return f"""Answer the question using only the context below.
If the context doesn't contain the answer, say so — don't guess.
Cite the source tag(s) you used, like [doc_id#chunk_index].

Context:
{context}

Question: {question}"""

Nothing here is specific to any particular model or framework. chunks is exactly the list store.search() returns in the vector databases post’s demo — this function is the one step the demo’s query.py stops short of.

Failure modes that are specific to this step

Retrieval failure caps generation quality. If the chunk that actually answers the question never makes it into the top-k, no amount of prompting fixes that — the model is reasoning over the wrong passages, or none at all. This is why the earlier posts spent so much time on chunk size and embedding choice: nothing downstream can undo a bad retrieval.

Context budget is a real constraint, not a suggestion. top_k chunks at chunk_size words each has to fit in the model’s context window alongside the prompt instructions and the eventual answer. Pushing top_k up to catch more possible answers pushes cost and latency up with it, and past a point, stuffing in more marginally-relevant chunks dilutes the ones that actually mattered.

Grounding is a strong nudge, not a guarantee. A model can still answer from its own memory instead of the supplied context, especially when the context is thin and the question touches something the model was confidently trained on. Explicit instructions (“only use the context below,” “say so if it’s not there”) help, but nothing about the architecture forces it — this is a prompting and evaluation problem, not something retrieval alone can close off.

An empty or irrelevant retrieval needs its own path. The same score_threshold idea from the vector databases post applies here at the generation step too: if nothing cleared the bar, that should reach the model as “no relevant context found,” not as three barely-related chunks presented like they’re the answer.

What’s past this

Two refinements sit between “retrieve top-k, generate” and a production RAG system, both deliberately out of scope here:

Reranking. Retrieve a wider set cheaply with the vector index, then run a slower, more accurate cross-encoder model over just those candidates to reorder them before the top few go into the prompt.
Hybrid search. Combine the vector index with ordinary keyword search (BM25) and merge the rankings, because semantic search alone still loses to keyword search on exact terms — product codes, names, error strings.

Both are ways of improving what lands in the chunks list above. The generation step itself doesn’t change.

Two things that bite beginners

A RAG system with bad retrieval looks like a hallucinating LLM. The symptom shows up at generation time, but the fix is almost always upstream — chunk size, embedding model choice, or the score threshold — not a better prompt.

More context isn’t automatically safer. It’s tempting to raise top_k “just in case.” Past a point that adds cost and dilutes the genuinely relevant chunks with noise, without measurably improving answers — the same lesson the chunking post drew about chunk size applies to how many chunks you hand the model at once.