You wouldn't deploy a web application without monitoring. So why are teams deploying LLM-powered features with zero observability?
At Black Gibbon, we use Arize AI as our primary LLM observability platform. After setting it up across eight client projects, here's our production-tested guide.
Why LLM Observability Matters
Traditional application monitoring tracks uptime, latency, and error rates. LLM observability adds three critical dimensions:
Response quality. Is the model's output accurate, relevant, and safe? Quality can degrade silently — the model still responds, but the answers get worse.
Cost tracking. Token usage directly impacts your bill. A prompt that balloons from 500 to 5,000 tokens due to a code change can 10x your API costs overnight.
Drift detection. User behavior changes over time. The queries your model handles in month three may be completely different from month one — and your prompts may need to adapt.
Setting Up Arize: Step by Step
Step 1: Instrument your LLM calls. Add Arize's SDK to every LLM API call. Log the prompt, response, latency, token count, and any metadata (user ID, feature flag, model version).
Step 2: Define quality metrics. For each LLM feature, define what "good" looks like. For a customer support bot, that might be: response relevance score above 0.8, no hallucinated policy information, and resolution within 3 turns.
Step 3: Set up embedding analysis. Arize can cluster your prompts by embedding similarity. This reveals prompt drift — when users start asking questions your system wasn't designed for.
Step 4: Configure alerts. Set thresholds for latency spikes, quality drops, cost anomalies, and new prompt clusters. We configure alerts to fire in Slack so the on-call engineer sees them immediately.
Step 5: Build dashboards. Create views for each stakeholder: engineering gets latency and error rates, product gets quality and usage patterns, finance gets cost breakdowns by feature.
Lessons from Production
After six months of running Arize across client projects, our biggest learning is this: the most valuable alert is the quality degradation alert. Latency and errors are obvious. Quality drops are invisible until customers complain.
One client's chatbot gradually shifted from answering questions using their knowledge base to generating responses from the LLM's training data. The responses sounded correct but contained outdated information. Without quality monitoring, this would have gone unnoticed for weeks.
Monitor your LLMs like you monitor your infrastructure. The failure modes are different, but the consequences are just as real.