Retrieval-Augmented Generation — RAG — has become the default architecture for enterprise AI features. Feed your company's documents into a vector database, retrieve relevant chunks when a user asks a question, and let an LLM generate an answer grounded in your actual data.
The pitch is compelling. The reality is messier.
We've built and fixed RAG systems for a dozen mid-market companies across Southern California. The pattern is always the same: the demo works perfectly, then production users start getting confident-sounding answers that are completely wrong. Here's why, and how to build RAG systems that actually work.
Why RAG Hallucinations Happen
Bad Chunking
The most common RAG failure has nothing to do with the AI model. It's bad chunking — splitting documents into pieces that lose their meaning.
A 50-page SOC 2 compliance document chunked into 500-token blocks will split sentences, separate tables from their headers, and divorce conclusions from the evidence that supports them. When the retriever pulls one of these orphaned chunks, the LLM fills in the gaps with plausible-sounding fiction.
Fix: Use semantic chunking that respects document structure. Split on section headers, paragraph boundaries, and logical units. Overlap chunks by 10–15% so context isn't lost at boundaries. For tables and lists, keep the entire structure in one chunk with its header.
Retrieval Misses
Vector similarity search isn't keyword search. A user asking "What's our refund policy?" might not match a document titled "Customer Returns and Exchanges Procedure" because the embedding spaces don't overlap cleanly.
When the retriever returns irrelevant chunks, the LLM has two bad options: admit it doesn't know (which most default prompts don't encourage), or generate an answer from its training data instead of your documents.
Fix: Implement hybrid search — combine vector similarity with keyword matching (BM25). Add query expansion using an LLM to rephrase the user's question into multiple search queries. And most importantly, set a relevance threshold — if no chunk scores above 0.7 similarity, return "I don't have enough information" instead of guessing.
Missing Metadata
A chunk that says "revenue increased 15% year over year" is useless without knowing which year, which division, and which document it came from. Without metadata, the LLM can't distinguish between Q1 2024 and Q3 2025 financials, between the West Coast and East Coast divisions, or between a draft and a final report.
Fix: Attach rich metadata to every chunk — source document, section, date, version, author, and document type. Include this metadata in the prompt so the LLM can cite its sources and the user can verify.
Architecture That Works
After building RAG systems across healthcare, fintech, and enterprise SaaS, here's the architecture we recommend for production:
Ingestion Pipeline
Documents go through a preprocessing pipeline: format conversion, OCR if needed, structure detection (headers, tables, lists), semantic chunking, metadata extraction, and embedding generation. This pipeline runs overnight in our Vietnam pod, processing new documents and re-indexing updated ones while the US team sleeps.
Dual Retrieval
Every query hits both a vector store (for semantic matching) and a keyword index (for exact matching). Results are merged and re-ranked using a cross-encoder model that scores each chunk's actual relevance to the query, not just its embedding similarity.
Answer Generation with Citations
The LLM receives the top 5–10 chunks with full metadata and explicit instructions: answer only from the provided context, cite the source document for every claim, and say "I don't have information on this" if the context doesn't cover the question.
Verification Layer
Before returning the answer to the user, a secondary LLM call checks whether each claim in the response is actually supported by the cited chunks. Claims that can't be verified are flagged or removed. This adds 1–2 seconds of latency but dramatically reduces hallucinations.
Feedback Loop
Every answer includes thumbs up/down buttons. Negative feedback triggers a human review of the query, retrieved chunks, and generated answer. These reviews feed back into chunking improvements, retrieval tuning, and prompt refinement.
The Metrics That Matter
Don't measure your RAG system by user satisfaction alone — satisfied users might be getting wrong answers they believe. Track these metrics:
Retrieval precision: What percentage of returned chunks are actually relevant to the query? Target: above 80%.
Answer faithfulness: What percentage of claims in the generated answer are supported by the retrieved chunks? Target: above 95%.
Answer coverage: What percentage of the information in the relevant chunks makes it into the answer? Target: above 70%.
Abstention rate: How often does the system correctly say "I don't know" instead of hallucinating? Higher is better than you think — a system that abstains 20% of the time is more trustworthy than one that always guesses.
RAG is not a weekend project. It's a production system that needs engineering discipline — chunking strategies, retrieval optimization, verification layers, and continuous monitoring. Build it right, and it becomes the most valuable AI feature your company ships.