We've interviewed a lot of "AI Engineers" over the past two years. We've also inherited a lot of what they built, usually after it caught fire in production and the original author had already moved on to their next six-month contract.
So let's just say it: most of the people with "AI Engineer" in their headline are prompt-and-pray developers who wired up a single chat completion call, watched it return something plausible on their laptop, and decided they'd leveled up. That's not a criticism of ambition. It's a warning about what happens when that person is the only thing standing between a language model and your customers.
We're not gatekeeping the title for fun. We're gatekeeping it because we've had to clean up the difference, and the cleanup is always more expensive than hiring the right person the first time.
Calling the API is the easy 5%
Here's the uncomfortable truth the wrapper crowd doesn't want to hear: the model call is the least interesting part of the job. Any decent backend engineer can send a prompt and parse the response in an afternoon. The framework docs will hold your hand through it.
The actual work is everything wrapped around that call. What happens when the model returns malformed JSON at 2am? What happens when a user pastes 40,000 tokens into a box you sized for 2,000? What happens when the provider has a partial outage and latency triples but requests still succeed, just slowly enough to blow your connection pool? What happens when the "helpful" answer is confidently, specifically wrong and a support rep acts on it?
A real AI engineer spends 90% of their time on those questions. The wrapper crowd spends 90% of their time tuning the prompt and calling it done.
We once took over a document-classification feature a contractor had shipped for a client. Looked great in the demo. In production it silently mislabeled about one in nine documents because the prompt asked for a category "as a single word" and the model occasionally answered with a short sentence. There was no validation. The code just took the first word of whatever came back. `"Not an invoice"` became the category `Not`. It ran that way for five weeks before anyone noticed the downstream reports were garbage.
That's not an AI problem. That's an engineering-discipline problem wearing an AI costume.
No evals means you're just vibing
Ask someone how they know their AI feature works. If the answer is "I tried a bunch of examples and it looked good," you're talking to a hobbyist, no matter what their title says.
You cannot ship a system whose behavior is non-deterministic and probabilistic and then verify it by eyeballing a handful of cases. That's not testing. That's a vibe. When you change a prompt, swap a model, or tweak your retrieval, you need to know whether you made things better or just moved the failures somewhere you didn't look.
Real AI engineers build eval sets. They collect the cases that actually broke, label them, and run every change against that set so a "small" prompt tweak doesn't quietly regress the twelve edge cases they fixed last month. We keep golden datasets per feature and gate deploys on them the same way we gate on unit tests. It's not glamorous. It's the reason our stuff doesn't degrade every time someone touches the prompt.
If your candidate has never built an eval harness, they've never actually maintained an AI feature over time. They've launched one and walked away.
Retrieval is where the real engineering lives
Everybody says "we'll just do RAG." Almost nobody thinks about retrieval as an engineering problem, which is exactly why so many RAG systems return confident nonsense.
The model is only as good as what you feed it, and "what you feed it" is a search problem, a chunking problem, a ranking problem, and a data-freshness problem all at once. We've seen teams dump raw PDFs into a vector store with default 512-token chunks, no overlap, no metadata, and then blame the model when it can't answer questions that span a page break. The model never had a chance. The relevant sentence got split down the middle and neither half retrieved.
A real AI engineer will interrogate your chunking strategy, ask whether you need hybrid search instead of pure vectors, think about how to handle documents that update daily, and know that reranking often buys more accuracy than swapping to a bigger, pricier model. They treat the LLM as the last step in a pipeline they actually engineered, not a magic box that fixes bad inputs.
Cost and latency are features, not afterthoughts
Here's a red flag that shows up on the invoice: an "AI engineer" who has no opinion about which model to use for which task.
We inherited a system routing every single request, including a trivial yes/no intent check, through the most expensive frontier model available. The bill was roughly $8,000 a month. We moved the cheap, high-volume classification work to a small fast model, kept the heavy reasoning on the expensive one, added a caching layer for repeated queries, and got it under $2,000 without any user-facing quality drop. Same product, quarter the cost, and it got faster because the small model responded in a fraction of the time.
That's not a magic trick. It's what happens when someone treats tokens and milliseconds as a budget they're accountable for instead of an externality. Ask a candidate how they'd cut an AI bill in half. If they only know "use a cheaper model," they don't understand the levers. If they mention routing, caching, prompt compression, batching, and knowing which calls can run async, keep talking to them.
The senior move is knowing when NOT to use an LLM
This is the one that separates the genuinely good from everyone else, and it's the most counterintuitive.
A lot of AI features shouldn't be AI features. We've watched people reach for an LLM to extract a date from a string, validate an email, or route a ticket based on three keywords. A regex, a lookup table, or a boring `if` statement would be faster, free, deterministic, and impossible to hallucinate. Wrapping a language model around a solved problem doesn't make you innovative. It makes you slow, expensive, and flaky.
The best AI engineers we've worked with are the ones who'll tell you, in the kickoff meeting, that half your proposed "AI" roadmap should just be normal code. They use the model where its judgment and language ability actually earn their keep, and they use deterministic logic everywhere else. That restraint is the whole job. Anyone can add an LLM to something. Knowing where it doesn't belong is the senior skill.
What we actually screen for
When we put an engineer on a client's production system, we're not checking whether they can call an API. We assume that. We're checking whether they'll write validation and fallbacks for every model output, whether they build evals before they ship, whether they think about retrieval as real infrastructure, whether they own the cost and latency budget, and whether they have the judgment to keep the LLM out of the places it doesn't belong. Then we put human QA and senior review behind all of it, because even great AI engineers ship bugs, and the model will happily help them do it.
That bar is why our stuff stays up after we hand it off. It's slower to hire for and it's harder to fake. That's the point.
If you've got an AI feature that demos beautifully and misbehaves in production, or a roadmap full of "AI" that you suspect is half regex, that's exactly the kind of thing our pod is built to sort out with you.