A finance lead at one of our clients asked their internal assistant a simple question: "What was our revenue growth last year?" The system answered, confidently, "Revenue increased 15% year over year." It cited a real internal document. It was also wrong.
Anatomy of One Hallucination
The 15% figure existed. It came from a draft Q1 2024 report covering only the West Coast division. The actual company-wide number for the year the user asked about was different, and it lived in a final report the system never retrieved. Nothing in the answer was invented from thin air — and that's exactly what made it dangerous. It looked grounded.
Trace it backward and three separate failures line up.
First, the chunk itself was orphaned. The sentence "revenue increased 15% year over year" had been split away from the header and table that qualified it — the division, the quarter, the "DRAFT" watermark. By the time the retriever saw it, it was a free-floating claim with no context attached.
Second, retrieval pulled that chunk instead of the right one. The user's phrasing ("revenue growth last year") sat closer in embedding space to the draft's language than to the final report, which used different wording. Vector similarity picked the confident-sounding match over the correct one.
Third, the model had no way to tell the two apart. With no metadata on the chunk — no date, no version, no division, no document type — the LLM couldn't distinguish a West Coast draft from a company-wide final. So it did what LLMs do when handed a plausible fragment and a question: it answered.
We've built and fixed RAG systems for a dozen mid-market companies across Southern California, and almost every hallucination we investigate decomposes the same way. The demo works perfectly, then production users start getting confident, well-sourced, wrong answers. The rest of this post is the practices that prevent each of those three failures — and the architecture and metrics that keep them prevented.
Fixing the Three Failures
Bad Chunking
The most common RAG failure has nothing to do with the AI model. It's bad chunking — splitting documents into pieces that lose their meaning.
A 50-page SOC 2 compliance document chunked into 500-token blocks will split sentences, separate tables from their headers, and divorce conclusions from the evidence that supports them. When the retriever pulls one of these orphaned chunks — like our 15% figure stripped from its "DRAFT" header — the LLM fills in the gaps with plausible-sounding fiction.
Fix: Use semantic chunking that respects document structure. Split on section headers, paragraph boundaries, and logical units. Overlap chunks by 10–15% so context isn't lost at boundaries. For tables and lists, keep the entire structure in one chunk with its header.
Retrieval Misses
Vector similarity search isn't keyword search. A user asking "What's our refund policy?" might not match a document titled "Customer Returns and Exchanges Procedure" because the embedding spaces don't overlap cleanly — the same mismatch that sent our finance query to a draft instead of the final report.
When the retriever returns irrelevant chunks, the LLM has two bad options: admit it doesn't know (which most default prompts don't encourage), or generate an answer from its training data instead of your documents.
Fix: Implement hybrid search — combine vector similarity with keyword matching (BM25). Add query expansion using an LLM to rephrase the user's question into multiple search queries. And most importantly, set a relevance threshold — if no chunk scores above 0.7 similarity, return "I don't have enough information" instead of guessing.
Missing Metadata
A chunk that says "revenue increased 15% year over year" is useless without knowing which year, which division, and which document it came from. Without metadata, the LLM can't distinguish between Q1 2024 and Q3 2025 financials, between the West Coast and East Coast divisions, or between a draft and a final report — the exact confusion behind the wrong answer above.
Fix: Attach rich metadata to every chunk — source document, section, date, version, author, and document type. Include this metadata in the prompt so the LLM can cite its sources and the user can verify.
Architecture That Works
After building RAG systems across healthcare, fintech, and enterprise SaaS, this is the architecture we recommend for production:
Ingestion Pipeline
Documents go through a preprocessing pipeline: format conversion, OCR if needed, structure detection (headers, tables, lists), semantic chunking, metadata extraction, and embedding generation. This pipeline runs overnight in our Vietnam pod, processing new documents and re-indexing updated ones while the US team sleeps.
Dual Retrieval
Every query hits both a vector store (for semantic matching) and a keyword index (for exact matching). Results are merged and re-ranked using a cross-encoder model that scores each chunk's actual relevance to the query, not just its embedding similarity.
Answer Generation with Citations
The LLM receives the top 5–10 chunks with full metadata and explicit instructions: answer only from the provided context, cite the source document for every claim, and say "I don't have information on this" if the context doesn't cover the question.
Verification Layer
Before returning the answer to the user, a secondary LLM call checks whether each claim in the response is actually supported by the cited chunks. Claims that can't be verified are flagged or removed. This adds 1–2 seconds of latency but dramatically reduces hallucinations — and would have caught the 15% claim, which no retrieved chunk supported at the company-wide level.
Feedback Loop
Every answer includes thumbs up/down buttons. Negative feedback triggers a human review of the query, retrieved chunks, and generated answer. These reviews feed back into chunking improvements, retrieval tuning, and prompt refinement.
The Metrics That Matter
Don't measure your RAG system by user satisfaction alone — satisfied users might be getting wrong answers they believe. Track these metrics:
Retrieval precision: What percentage of returned chunks are actually relevant to the query? Target: above 80%.
Answer faithfulness: What percentage of claims in the generated answer are supported by the retrieved chunks? Target: above 95%.
Answer coverage: What percentage of the information in the relevant chunks makes it into the answer? Target: above 70%.
Abstention rate: How often does the system correctly say "I don't know" instead of hallucinating? Higher is better than you think — a system that abstains 20% of the time is more trustworthy than one that always guesses.
RAG is not a weekend project. It's a production system that needs engineering discipline — chunking strategies, retrieval optimization, verification layers, and continuous monitoring. Build it right, and it becomes the most valuable AI feature your company ships.