LLM Observability and Evaluation: Traces, Quality Metrics, and Experiments

Start With a Quality Regression Nobody Noticed

Your RAG support assistant has been running smoothly for three weeks. Latency is stable at 2 seconds. Error rate is under 0.5%. The team is focused on building new features.

On day 21, a user posts on social media:

"Your AI assistant told me I can cancel my annual subscription for a full refund. That is wrong. Your policy clearly says annual subscriptions are non-refundable."

You check the monitoring dashboard. Latency: green. Error rate: green. Everything looks fine. But the answer was wrong.

What happened? Three days ago, a teammate updated the system prompt from "Answer concisely" to "Be thorough and helpful." A new model version was rolled out by the provider. And the retrieval top-k was changed from 5 to 10 in a config deploy.

Each change seemed harmless. Each was logged somewhere. But no system connected the dots: prompt change + model update + retrieval change = worse answers. The observability system tracked uptime, not quality.

An LLM call can return HTTP 200 and still be a bad answer. Traditional observability — latency, errors, and saturation — is necessary but not sufficient for LLM systems. You also need to track what the model was asked, what it was given, what it produced, how much it cost, and whether the answer was any good.

Why LLM Observability Is Different

Traditional web service observability tracks three pillars: logs, metrics, and traces. For an API endpoint, you measure request rate, latency percentiles, and error codes. If the endpoint returns 200 in under 200ms, everything is fine.

LLM observability needs all of that plus:

Why Not Just Use Application Logs?

Beginners often ask: "Can I just log the prompt and response to my existing logging system?"

You can, but LLM observability has requirements that general-purpose logging does not satisfy.

Step 1: Tracing the Full Pipeline

Tracing in LLM systems means capturing every step from user request to final response as connected spans. Each span captures timing, inputs, outputs, and metadata.

Span Schema

Every span should capture:

{
  "spanId": "span_abc123",
  "traceId": "trace_def456",
  "parentSpanId": "span_parent789",
  "service": "rag-pipeline",
  "spanName": "retrieval",
  "startTime": "2026-06-01T10:00:00.000Z",
  "duration": 320,
  "attributes": {
    "query": "Can I pause my annual subscription?",
    "topK": 10,
    "filter": "tenant=acme",
    "documentIds": ["doc_001", "doc_042", "doc_103"],
    "scores": [0.92, 0.87, 0.64],
    "indexVersion": "idx_2026_05_28"
  }
}

Which Spans to Capture

Sampling Strategy

Storing every trace for every request is expensive. Use a sampling strategy:

Step 2: Metrics — Reliability, Cost, and Quality

Metrics are aggregated measurements over time. LLM systems need three categories of metrics.

Reliability and Performance

Cost Metrics

Cost per request calculation:
  Cost = (input_tokens × input_price_per_token)
       + (output_tokens × output_price_per_token)
       + retrieval_cost
       + reranking_cost

Example for a RAG request:
  Input: 3500 tokens × $3/M tokens = $0.0105
  Output: 450 tokens × $15/M tokens = $0.0068
  Retrieval: 1 vector search = $0.0002
  Reranking: 20 candidates reranked = $0.0005
  Total: $0.018 per request

  At 100K requests/day: $1,800/day = $54,000/month
  A 10% context reduction saves $5,400/month

Quality Metrics

Step 3: RAG Evaluation

RAG has two subsystems that fail differently. Retrieval can miss relevant documents. Generation can misinterpret or ignore the retrieved evidence. They must be evaluated separately.

Retrieval Evaluation Metrics

Generation Evaluation Metrics

Building a Golden Dataset

Question: "Can I pause my annual subscription?"
Expected relevant docs: ["billing-faq.md#annual-pause", "subscription-policy.pdf#section-4.2"]
Expected answer: "Annual subscriptions cannot be paused. Customers on annual plans can contact
  support for a credit review if there was a billing mistake or exceptional case."
Expected citations: [billing-faq.md#annual-pause]
Expected confidence: high
Acceptable variants: Mentions credit review, directs to support, does not promise pause
Unacceptable: Says annual subscriptions can be paused, does not mention exception path

Question: "What is the refund policy for enterprise customers?"
Expected relevant docs: ["enterprise-agreement-template.pdf#refund-clause"]
Expected answer: "Enterprise refunds are governed by the terms in your specific agreement.
  Please contact your account manager for details."
Expected citations: [enterprise-agreement-template.pdf#refund-clause]
Expected behavior: Should NOT guess specific refund terms, should direct to account manager

Dataset Requirements

Step 4: LLM-as-Judge

LLM-as-judge uses a separate model to evaluate the quality of your primary model's outputs. It scales evaluation to thousands of examples but introduces its own biases.

What to Judge

Judge Prompt Template

You are evaluating a support assistant's response.

User question: {question}
Retrieved context: {context}
Assistant response: {answer}

Rate the following dimensions on a scale of 1-5:

1. Relevance: Does the answer address the user's question?
2. Faithfulness: Are all claims supported by the context?
3. Completeness: Does the answer cover all aspects of the question?

Provide a score and a brief rationale for each dimension.
If any claim in the answer is NOT supported by the context, mark faithfulness as 1.

Risks and Calibration

Calibration with Human Labels

Human-labeled example 1:
  Question: "Can I pause my annual subscription?"
  Context: [billing-faq: annual subscriptions cannot be paused]
  Answer: "No, annual subscriptions cannot be paused. You can contact support for a credit review."
  Human score: relevance=5, faithfulness=5
  Judge score: relevance=5, faithfulness=5 ✓

Human-labeled example 2:
  Question: "What is my account number?"
  Context: [no context contains the user's account number]
  Answer: "Your account number is ACC-784512."
  Human score: faithfulness=1 (hallucination, not in context)
  Judge score: faithfulness=5 ✗ (judge failed to detect hallucination)

If the judge consistently disagrees with human labels on certain criteria, adjust the judge prompt or switch to a different judge model.

Step 5: A/B Testing and Experiments

LLM systems change constantly: prompts, models, retrieval parameters, rerankers, tools, and guardrails. Every change is a potential regression. A/B testing measures whether the change actually improves quality, cost, or latency.

What to Test

Experiment Design

Experiment Requirements

Statistical Significance

Before declaring a winner, verify:
- At least 1000 samples per variant
- The difference in key metrics is statistically significant (p < 0.05)
- No metric degraded significantly (even if primary metric improved)
- The effect is consistent across tenants and user segments
- The experiment ran through at least one full business cycle

Example result:
  Variant A (control):  85% quality score, $0.018/request, 2.1s latency
  Variant B (treatment): 91% quality score, $0.022/request, 2.4s latency

  Decision: Quality improved by 6 points (p < 0.01). Cost increased 22%.
  Latency increased 14%. Deploy if the quality improvement justifies
  the cost increase. Revert if not.

Step 6: Prompt and Model Versioning

Every production response should be traceable to the exact versions of every artifact that produced it. Without versioning, you cannot debug regressions.

Traceable Response Metadata

Every response should carry version metadata that links back to the artifacts that produced it:

{
  "response": "Annual subscriptions cannot be paused...",
  "metadata": {
    "promptVersion": "v2.4.1",
    "promptHash": "a1b2c3d4",
    "modelName": "gpt-4o",
    "modelVersion": "2026-05-15",
    "embeddingModel": "text-embedding-3-small@002",
    "retrievalConfig": "retrieval_config_v3.json",
    "indexVersion": "idx_2026_05_28",
    "guardrailPolicy": "guardrails_v2.json",
    "requestTimestamp": "2026-06-01T10:00:00Z"
  }
}

Versioning Workflow

Step 7: Alerting and Dashboards

Alerts should catch regressions before users notice them.

Alert Rules

Dashboard Design

Organize dashboards by audience:

Operations Dashboard (on-call engineers):

• Request rate, error rate, latency (P50/P95) over time
• Provider status and error breakdown
• Cost per hour
• Active experiments

Quality Dashboard (ML engineers):

• LLM-as-judge quality scores over time by dimension
• Retrieval recall and precision trends
• Golden dataset pass/fail rate per version
• User feedback (thumbs up/down) over time

Business Dashboard (product managers):

• Cost per feature and per tenant
• Task success rate
• User satisfaction score
• Monthly cost trend vs budget
• Experiment results and recommendations

Common Failure Stories

The Prompt Changed But Nobody Logged It

A developer edits the system prompt directly in the production configuration file. No version bump. No pull request. No golden dataset run. Three days later, quality metrics drop. The team spends a week debugging retrieval, model, and infrastructure before someone notices the prompt changed.

The fix: prompts must be versioned and deployed through the same CI/CD pipeline as code. Every prompt change should trigger the golden dataset evaluation suite. If quality drops, the deployment should block.

The Model Update Broke Citations

The provider releases a new model version that changes the output format. The structured output schema still validates, but the model stops including citations. The validation layer passes because the schema allows empty citation arrays. For three days, the system serves uncited answers.

The fix: set a minimum citation count in the validation schema. If the model produces fewer than the required number of citations, the validation should fail and trigger an alert. Never rely on the model alone to follow formatting instructions.

The A/B Test Was Inconclusive Because of Traffic Skew

An experiment splits traffic 50/50 by user ID hash. But Variant B gets 80% of the traffic from enterprise tenants with complex questions. Variant B looks worse because it handles harder questions, not because the change is bad.

The fix: stratify experiment splits by tenant type, question complexity, and feature. Measure metrics per segment, not just aggregated. Or use a true randomized assignment within each segment.

The LLM-as-Judge Missed a Domain-Specific Error

A medical question about drug interactions gets a plausible-sounding but wrong answer. The judge model, also a general-purpose LLM, does not have enough medical knowledge to detect the error. It scores faithfulness as 5 because the answer sounds reasonable.

The fix: for high-stakes domains, use a domain-specific judge model or include domain rules in the judge prompt. Better yet, do not rely solely on LLM-as-judge for domains where errors have serious consequences. Use human review for a sample.

The Golden Dataset Did Not Catch the Regression

The team updates the retrieval config from top-5 to top-10. The golden dataset passes because all expected documents appear in the top-10. But the extra 5 documents add noise, and the model starts hallucinating from irrelevant context. The quality score drops in production but the golden dataset still passes.

The fix: the golden dataset should measure not just retrieval recall but also generation quality. Add test cases where irrelevant context is present, and measure whether the model correctly ignores it. The evaluation should mirror the full production pipeline, not just individual components.

Evaluating the Observability System

The observability system itself needs evaluation. Can it actually detect regressions?

A Complete Observability Pipeline, End to End

Here is how observability data flows from a single request to actionable insight:

This flow connects every component: tracing captures what happened, metrics quantify it, evaluation scores quality, versioning provides context, and alerts surface regressions. Without any one of these pieces, you are flying blind.

What to Remember for Interviews

When explaining LLM observability, tell the story in order:

1. Traditional observability is not enough: HTTP 200 with a bad answer is a successful call and a failed interaction. LLM systems need quality signals, not just uptime signals.
2. Trace the full pipeline: Every span — retrieval, reranking, prompt assembly, LLM call, validation — must be connected by a trace ID. Sample happy paths, store 100% of failures.
3. Track three categories of metrics: Reliability (latency, error rate), cost (tokens, spend per feature), and quality (relevance, faithfulness, citation accuracy).
4. Evaluate retrieval and generation separately: They fail differently. Use a golden dataset with real user questions, expected sources, and acceptable answer criteria.
5. Use LLM-as-judge carefully: It scales evaluation but has biases (self-enhancement, position, length, domain blindness). Calibrate against human labels. Recalibrate regularly.
6. Version everything: Prompts, models, embedding models, retrieval configs, guardrail policies. Every response should carry the versions that produced it.
7. A/B test every change: Prompt edits, model upgrades, retrieval changes — measure quality, cost, and latency simultaneously. A quality improvement that doubles cost may not be worth it.
8. Alert on quality, not just errors: Cost spikes, quality drops, no-citation rates, and validation failures are the signals that matter for LLM systems.