The shipment assistant answered everything fluently. It was also wrong about one time in five — and a customer-facing tool that's confidently wrong 20% of the time is a tool you end up switching off.
By the time the SoCal logistics company called us, they'd already spent six weeks on the model: new foundation model, new prompts, new embeddings, a different vector store. We never touched the model. We spent four days in the data — three sources feeding one assistant, a Postgres orders table, a warehouse management system exporting CSVs, and a third-party carrier API — and came back with three culprits.
- The carrier API returned timestamps in UTC. The WMS exported in America/Los_Angeles with no timezone marker. Postgres stored `created_at` in UTC but `promised_date` in local time. So "is this late" was comparing apples to a clock that was seven or eight hours off, depending on daylight saving.
Three bugs, zero of them in the model. That's the pattern under almost every AI feature that ships fluent and wrong: the model isn't the bottleneck — the data feeding it is. "Garbage in, garbage out" isn't a cliché you've outgrown. A large language model is an expensive amplifier, and an amplifier makes bad input louder, not better. The rest of this is the unglamorous engineering that keeps the garbage out — the part nobody puts in the demo.
Ingestion is where the lies get in, so validate at the door.
The cheapest place to catch bad data is the moment it arrives. The most expensive place is three transformations downstream when it's already mixed into everything.
Treat every source as hostile until proven otherwise. When we ingest, we validate at the boundary before a single row lands in the warehouse: row counts against the previous load (a file that's suddenly 4% of yesterday's size is a truncated export, not a slow day), required columns present, types coercible, and a handful of business invariants (a ship date can't precede an order date). We quarantine what fails instead of letting it flow through. A rejected batch that pages a human beats a silently half-loaded table that poisons a model's answers for a week.
Practical rule: if you can't say out loud what a valid record looks like, you're not ready to ingest it. Write that definition down as code, not as tribal knowledge.
Cleaning and normalization is 60% of the job, and it's boring on purpose.
The unglamorous middle is where features are won. Some of what we do on nearly every engagement:
- Encodings. Real-world text is a mess of UTF-8, Latin-1, and Windows-1252 pretending to be each other. That's where `José` becomes `José`. Normalize to UTF-8 at ingestion and Unicode-normalize (NFC) so that visually identical strings compare equal.
This work isn't clever. It's disciplined. But it's the difference between retrieval that finds the right records and retrieval that finds most of them.
Schemas and data contracts are how you stop breaking in production at 2 a.m.
The single most common way an AI feature silently degrades: an upstream team renames a column, changes a unit, or starts sending nulls in a field that was never null, and nobody tells you. Your pipeline keeps running. Your outputs quietly rot.
A data contract is an explicit, enforced agreement about the shape and meaning of what flows between systems: column names, types, nullability, units, allowed value ranges, and a promise not to change them without a version bump. Enforce it in code with a schema validator on every load. When production data violates the contract, you want a loud failure and a quarantined batch, not a model confidently reporting freight weights in kilograms because someone upstream switched units.
PII has to be handled at the pipeline layer, not bolted on later.
Once messy data is flowing, you're also flowing customer names, emails, addresses, and sometimes worse into logs, embeddings, and model context. Handle it where the data lives. Classify PII fields at ingestion, mask or tokenize what the AI feature doesn't strictly need, and keep it out of your logs and out of embeddings you can't easily delete. Retrofitting privacy after you've vectorized a million records containing raw email addresses is a bad week you can avoid with a day of upfront design.
Lineage and monitoring are the difference between a bug and a mystery.
When an AI feature gives a wrong answer, the first question is "where did this come from?" If you can't trace an output back through the transformations to the source row, you're debugging blind. Track lineage so any field can be walked back to its origin. And monitor data quality continuously, not just at launch: null rates, row-count deltas, value distributions, freshness (is this source even updating?), and dedup rates. Alert on drift. The carrier's timezone bug above would have been caught on day one by a simple check that flagged when "on-time" rates shifted by a suspicious amount overnight.
The checklist to keep on your desk.
Before you blame the model, walk this list:
1. Do you validate every source at ingestion and quarantine failures?
If you can't confidently check most of those, your AI feature's problem isn't the model.
When you'd rather someone just did this.
This is the work our data engineers do embedded alongside your team, out of Irvine and our Vietnam bench. Real pipelines, real production systems, human-in-the-loop from ingestion to monitoring. If your AI feature is fluent but wrong, and you suspect the data underneath it is the reason, that's exactly the fix we ship. Rent a data engineer from our pod and let's find your garbage before your customers do.