Everyone builds a RAG in an afternoon. You drop documents into a vector store, run a similarity search, stuff the chunks into a prompt, and the model answers. The demo works, the stakeholder smiles, and you think you shipped. Three weeks later, with real users, half the answers are wrong, your embedding bill has tripled, and nobody can explain why "what's the warranty period?" returns the return policy. These are the notes I wish I'd read before making those mistakes in production.
RAG is not an LLM problem, it's a retrieval problem
The most expensive intuition I've watched teams carry is treating RAG as "prompt engineering with context." The LLM is the easy, nearly commoditized part. What separates a decent system from a reliable one is the quality of what lands in the context window. If retrieval brought back the wrong passage, no elegant prompt saves the answer — the model will confidently hallucinate about the wrong document. I started measuring my systems by retrieval before looking at generation, and it changed how I allocate engineering time.
Chunking is where the project lives or dies
The classic mistake is splitting documents into fixed-size windows — 512 tokens, no overlap, structure ignored. The result is a table cut in half, a list item that loses the heading that gave it meaning, and the one sentence that answers the question squeezed between two chunks so neither is retrieved whole.
What worked best for me was structure-aware chunking: respect semantic boundaries (sections, paragraphs, table cells) before cutting by size, keep a 10–15% overlap, and — crucially — attach to each chunk the metadata of where it came from: document title, section, date. That metadata isn't decoration: it becomes filters at search time and extra context in the prompt. For long, hierarchical documents, indexing both a top-level summary and the detailed passages (the "parent-child" pattern) solved most cases where the answer depended on context two sections up.
Pure vector search is a trap
Dense embeddings are excellent at capturing semantic similarity and terrible at exact matching. The day a user searches for the error code "ERR_4012" or a SKU name, vector search abandons you — those rare tokens have no useful semantic neighborhood. The solution that became standard in my projects is hybrid retrieval: combine dense search (embeddings) with sparse lexical search (BM25), and fuse the rankings with Reciprocal Rank Fusion. BM25 catches the literal identifier; the vector catches the paraphrase. Together they cover what each alone lets through.
Even with hybrid, the step that raised my precision the most was reranking. Retrieve 30–50 candidates with the cheap search, then reorder them with a cross-encoder and keep only the top 4–6 — it's the best return per dollar I know of in RAG. The first-stage retriever optimizes recall; the reranker optimizes precision. Trying to make the first-stage retriever precise enough to skip the reranker is optimizing the wrong thing.
Context is not free
There's a naive belief that "more context is better." It isn't. There's the obvious cost — you pay per input token, and cramming 20 chunks into the prompt multiplies the bill on every call. And there's the silent cost: lost-in-the-middle degradation. Models pay less attention to what's buried in the middle of long windows. Ten mediocre chunks make the answer worse than four precise ones, while costing more and adding latency. I treat the context budget as a scarce resource to be defended, not filled.
If you don't evaluate, you're guessing
The most neglected part, and the one that most separates amateur from engineer: evaluation. "It looks good when I test it" is not a metric. Before any optimization, I built a golden set of 100–200 question/answer pairs drawn from real user questions. From it I measure two separate layers:
- Retrieval: is the correct chunk among those retrieved? Metrics like recall@k and hit rate. This isolates whether the problem is search.
- Generation: given correct retrieval, is the answer faithful to the source? Here I use faithfulness (is the answer supported by the context?) and relevance, often with an LLM as judge against an explicit rubric.
Separating the two layers is what makes debugging possible. When an answer is bad, the first question is always: was it retrieval or generation? Without that split, you spend days tuning the prompt when the real problem was chunking.
The costs that show up after deploy
Re-indexing the whole corpus because you swapped the embedding model is expensive and slow — version your embeddings and plan incremental migrations. Latency is the sum of embedding the query, searching, reranking, and generating; each stage is a place for a timeout to hit you in production. Caching embeddings of frequent queries and answers to repeated questions cut my bill materially. And the knowledge base rots: documents change, policies get updated, and a RAG pointing at the stale version is worse than no RAG, because it's wrong with the authority of a citation.
What I'd do again on day one
I'd start with evaluation, not generation. I'd build the golden set before writing the first prompt. I'd use hybrid retrieval with reranking from the start instead of bolting them on later as a patch. I'd treat chunking and metadata as architecture decisions, not implementation details. And above all, I'd remember that in a RAG the model is the guest — retrieval runs the party.