Everyone wants to talk about the model. Which foundation model, which prompt, which fine-tune, which vector store. We get on calls where a team has spent six weeks A/B testing prompts and swapping embedding models, and the actual problem is that 12% of their input records have dates stored as strings like `03/04/05` with no century and no consistent order. The model was never the bottleneck. The data feeding it was.
This is a field guide to the part nobody puts in the demo. If you take one thing from it: the phrase "garbage in, garbage out" is not a cliché you've outgrown. It's still the entire game. A large language model is a very expensive amplifier, and an amplifier makes your bad inputs louder, not better.
The model was fine. The data was the whole problem.
Here's a real shape of project we see constantly, with the details filed off. A SoCal logistics company wanted an AI assistant that answered "where is my shipment and why is it late" from their internal systems. They'd wired up a solid retrieval setup over three data sources: a Postgres orders table, a warehouse management system exporting CSVs, and a third-party carrier API. The model was answering fluently. It was also wrong about 20% of the time, confidently, and that number is a business killer for a customer-facing tool.
We spent zero time on the model. We spent four days on the data and found this:
- The carrier API returned timestamps in UTC. The WMS exported in America/Los_Angeles with no timezone marker. Postgres stored `created_at` in UTC but `promised_date` in local time. So "is this late" was comparing apples to a clock that was seven or eight hours off, depending on daylight saving.
None of that is an AI problem. Every one of those is a data-engineering problem, and until you fix them, a better model just gives you more articulate wrong answers.
Ingestion is where the lies get in, so validate at the door.
The cheapest place to catch bad data is the moment it arrives. The most expensive place is three transformations downstream when it's already mixed into everything.
Treat every source as hostile until proven otherwise. When we ingest, we validate at the boundary before a single row lands in the warehouse: row counts against the previous load (a file that's suddenly 4% of yesterday's size is a truncated export, not a slow day), required columns present, types coercible, and a handful of business invariants (a ship date can't precede an order date). We quarantine what fails instead of letting it flow through. A rejected batch that pages a human beats a silently half-loaded table that poisons a model's answers for a week.
Practical rule: if you can't say out loud what a valid record looks like, you're not ready to ingest it. Write that definition down as code, not as tribal knowledge.
Cleaning and normalization is 60% of the job, and it's boring on purpose.
The unglamorous middle is where features are won. Some of what we do on nearly every engagement:
- Encodings. Real-world text is a mess of UTF-8, Latin-1, and Windows-1252 pretending to be each other. That's where `José` becomes `José`. Normalize to UTF-8 at ingestion and Unicode-normalize (NFC) so that visually identical strings compare equal.
This work isn't clever. It's disciplined. But it's the difference between retrieval that finds the right records and retrieval that finds most of them.
Schemas and data contracts are how you stop breaking in production at 2 a.m.
The single most common way an AI feature silently degrades: an upstream team renames a column, changes a unit, or starts sending nulls in a field that was never null, and nobody tells you. Your pipeline keeps running. Your outputs quietly rot.
A data contract is an explicit, enforced agreement about the shape and meaning of what flows between systems: column names, types, nullability, units, allowed value ranges, and a promise not to change them without a version bump. Enforce it in code with a schema validator on every load. When production data violates the contract, you want a loud failure and a quarantined batch, not a model confidently reporting freight weights in kilograms because someone upstream switched units.
PII has to be handled at the pipeline layer, not bolted on later.
Once messy data is flowing, you're also flowing customer names, emails, addresses, and sometimes worse into logs, embeddings, and model context. Handle it where the data lives. Classify PII fields at ingestion, mask or tokenize what the AI feature doesn't strictly need, and keep it out of your logs and out of embeddings you can't easily delete. Retrofitting privacy after you've vectorized a million records containing raw email addresses is a bad week you can avoid with a day of upfront design.
Lineage and monitoring are the difference between a bug and a mystery.
When an AI feature gives a wrong answer, the first question is "where did this come from?" If you can't trace an output back through the transformations to the source row, you're debugging blind. Track lineage so any field can be walked back to its origin. And monitor data quality continuously, not just at launch: null rates, row-count deltas, value distributions, freshness (is this source even updating?), and dedup rates. Alert on drift. The carrier's timezone bug above would have been caught on day one by a simple check that flagged when "on-time" rates shifted by a suspicious amount overnight.
The checklist to keep on your desk.
Before you blame the model, walk this list:
1. Do you validate every source at ingestion and quarantine failures?
If you can't confidently check most of those, your AI feature's problem isn't the model.
When you'd rather someone just did this.
This is the work our data engineers do embedded alongside your team, out of Irvine and our Vietnam bench. Real pipelines, real production systems, human-in-the-loop from ingestion to monitoring. If your AI feature is fluent but wrong, and you suspect the data underneath it is the reason, that's exactly the fix we ship. Rent a data engineer from our pod and let's find your garbage before your customers do.