Setting Up Arize for LLM Observability: A Production Guide

This is the runbook we follow to stand up Arize AI observability on a production LLM feature. We've run it across eight client projects. Work through the steps in order.

Prerequisites

Before you instrument anything, have three things in place:

- An Arize account and API keys for the workspace you're logging to.

Access to every code path that makes an LLM API call — you'll be adding logging at each one.

A definition of what "good" output means for each feature. You'll need this in Step 2, so decide it up front. For a customer support bot, that might be: response relevance score above 0.8, no hallucinated policy information, and resolution within 3 turns.

Step 1: Instrument your LLM calls

Add Arize's SDK to every LLM API call. Log the prompt, response, latency, token count, and any metadata (user ID, feature flag, model version). Miss a call path here and it stays invisible for the rest of the setup, so instrument all of them before moving on.

Step 2: Define quality metrics

For each LLM feature, encode what "good" looks like using the definitions from your prerequisites. This is what turns raw logs into a signal you can alert on. Response quality can degrade silently — the model still responds, but the answers get worse — so make these metrics explicit rather than eyeballing outputs later.

Step 3: Set up embedding analysis

Configure Arize to cluster your prompts by embedding similarity. This reveals prompt drift — when users start asking questions your system wasn't designed for. User behavior changes over time: the queries your model handles in month three may be completely different from month one, and your prompts may need to adapt. Embedding clusters are how you see that shift coming.

Step 4: Configure alerts

Set thresholds for latency spikes, quality drops, cost anomalies, and new prompt clusters. Wire alerts to fire in Slack so the on-call engineer sees them immediately.

Watch cost here specifically. Token usage directly impacts your bill, and a prompt that balloons from 500 to 5,000 tokens due to a code change can 10x your API costs overnight. A cost-anomaly alert catches that the same day instead of at the end of the billing cycle.

Step 5: Build dashboards

Create a view for each stakeholder: engineering gets latency and error rates, product gets quality and usage patterns, finance gets cost breakdowns by feature. Traditional application monitoring already tracks uptime, latency, and error rates — these dashboards add the quality, drift, and cost dimensions on top.

What we learned running this in production

After six months of running Arize across client projects, our biggest learning is this: the most valuable alert is the quality degradation alert. Latency and errors are obvious. Quality drops are invisible until customers complain.

One client's chatbot gradually shifted from answering questions using their knowledge base to generating responses from the LLM's training data. The responses sounded correct but contained outdated information. Without quality monitoring, this would have gone unnoticed for weeks.

Monitor your LLMs like you monitor your infrastructure. The failure modes are different, but the consequences are just as real.