The category was `Not`.
A contractor had shipped a document-classification feature for a client, and we took it over after the fact. It demoed beautifully. In production it silently mislabeled about one in nine documents, because the prompt asked for a category "as a single word" and the model occasionally answered with a short sentence. There was no validation. The code just took the first word of whatever came back. `"Not an invoice"` became the category `Not`. It ran that way for five weeks before anyone noticed the downstream reports were garbage.
The person who wrote it had "AI Engineer" in their headline and had already moved on to their next six-month contract. That's not an AI problem. It's an engineering-discipline problem wearing an AI costume, and it's the exact failure our hiring bar exists to catch.
So, the bar. When we put an engineer on a client's production system, we're not checking whether they can call an API. We assume that. We check for five specific things.
They treat the API call as the easy 5%
The model call is the least interesting part of the job. Any decent backend engineer can send a prompt and parse the response in an afternoon; the framework docs will hold your hand through it. The actual work is everything wrapped around that call.
What happens when the model returns malformed JSON at 2am? When a user pastes 40,000 tokens into a box you sized for 2,000? When the provider has a partial outage and latency triples but requests still succeed, just slowly enough to blow your connection pool? When the "helpful" answer is confidently, specifically wrong and a support rep acts on it?
A real AI engineer spends 90% of their time on those questions, writing validation and fallbacks for every model output. The wrapper crowd spends 90% of their time tuning the prompt and calling it done. The `Not` bug is what the second group ships.
They build evals, not vibes
Ask someone how they know their AI feature works. If the answer is "I tried a bunch of examples and it looked good," you're talking to a hobbyist, no matter what their title says.
You cannot ship a system whose behavior is non-deterministic and probabilistic and then verify it by eyeballing a handful of cases. That's not testing. That's a vibe. When you change a prompt, swap a model, or tweak your retrieval, you need to know whether you made things better or just moved the failures somewhere you didn't look.
Real AI engineers collect the cases that actually broke, label them, and run every change against that set so a "small" prompt tweak doesn't quietly regress the twelve edge cases they fixed last month. We keep golden datasets per feature and gate deploys on them the same way we gate on unit tests. If a candidate has never built an eval harness, they've never actually maintained an AI feature over time. They launched one and walked away.
They treat retrieval as real engineering
Everybody says "we'll just do RAG." Almost nobody thinks about retrieval as an engineering problem, which is exactly why so many RAG systems return confident nonsense.
The model is only as good as what you feed it, and "what you feed it" is a search problem, a chunking problem, a ranking problem, and a data-freshness problem all at once. We've seen teams dump raw PDFs into a vector store with default 512-token chunks, no overlap, no metadata, and then blame the model when it can't answer questions that span a page break. The model never had a chance. The relevant sentence got split down the middle and neither half retrieved.
A real AI engineer will interrogate your chunking strategy, ask whether you need hybrid search instead of pure vectors, think about how to handle documents that update daily, and know that reranking often buys more accuracy than swapping to a bigger, pricier model. They treat the LLM as the last step in a pipeline they actually engineered, not a magic box that fixes bad inputs.
They own the cost and latency budget
A red flag that shows up on the invoice: an "AI engineer" who has no opinion about which model to use for which task.
We inherited a system routing every single request, including a trivial yes/no intent check, through the most expensive frontier model available. The bill was roughly $8,000 a month. We moved the cheap, high-volume classification work to a small fast model, kept the heavy reasoning on the expensive one, added a caching layer for repeated queries, and got it under $2,000 without any user-facing quality drop. Same product, quarter the cost, and it got faster because the small model responded in a fraction of the time.
That's what happens when someone treats tokens and milliseconds as a budget they're accountable for instead of an externality. Ask a candidate how they'd cut an AI bill in half. If they only know "use a cheaper model," they don't understand the levers. If they mention routing, caching, prompt compression, batching, and knowing which calls can run async, keep talking to them.
They know when NOT to use an LLM
This is the one that separates the genuinely good from everyone else, and it's the most counterintuitive.
A lot of AI features shouldn't be AI features. We've watched people reach for an LLM to extract a date from a string, validate an email, or route a ticket based on three keywords. A regex, a lookup table, or a boring `if` statement would be faster, free, deterministic, and impossible to hallucinate. Wrapping a language model around a solved problem doesn't make you innovative. It makes you slow, expensive, and flaky.
The best AI engineers we've worked with will tell you, in the kickoff meeting, that half your proposed "AI" roadmap should just be normal code. They use the model where its judgment and language ability actually earn their keep, and they use deterministic logic everywhere else. Anyone can add an LLM to something. Knowing where it doesn't belong is the senior skill.
Why the bar is this high
Those five criteria are what we screen for before anyone touches a client's production system: validation and fallbacks on every output, evals before shipping, retrieval as real infrastructure, ownership of the cost and latency budget, and the judgment to keep the LLM out of the places it doesn't belong. Then we put human QA and senior review behind all of it, because even great AI engineers ship bugs, and the model will happily help them do it.
That bar is why our stuff stays up after we hand it off. It's slower to hire for and it's harder to fake. That's the point. It's also the difference between a feature that demos beautifully and one that doesn't spend five weeks labeling documents `Not` while everyone downstream trusts the numbers.