The pitch for RAG sounds simple: embed your documents, store the vectors, retrieve the relevant ones at query time, feed them to the LLM. Five steps. Clean.

Then you build it and discover that everything in those five steps has a dozen wrong ways to do it.

Where most RAG implementations go wrong

The most common mistake I see in RAG tutorials is treating all text as equal. Chunking a PDF into fixed 512-token blocks and embedding them all the same way produces a retrieval system that will confidently return the wrong section of a document — especially for multi-section documents like contracts, technical specs, or research papers.

The three things that actually matter

• Chunking strategy — Semantic chunking (split at natural document boundaries like headings) consistently outperforms fixed-size chunking for structured documents.
• Embedding model quality — Voyage AI's voyage-3 and OpenAI's text-embedding-3-large produce meaningfully better retrieval than older models. Do not skip this evaluation.
• Reranking — Retrieve more candidates (top-20), then rerank to top-5 before sending to the LLM. A cheap cross-encoder reranker catches cases where cosine similarity fails.

What I built: ContextChat

ContextChat uses Voyage AI embeddings + Upstash Vector for storage + Claude for the final answer. The retrieval pipeline: semantic chunks → embed → store. At query time: embed query → retrieve top-20 → Claude reranks to top-5 → final answer with citations back to source pages.

The citations are the feature that makes it trustworthy — users can verify every claim against the original document.