Grafana Dashboards: From Metrics to an Actual Dashboard

Metrics Nobody Looks At

/actuator/prometheus has been scrapeable since the last guide. Nobody at Acme Fintech has looked at it once — raw Prometheus output is not something a support-team lead or an on-call engineer opens voluntarily. This guide is the last step: turning those metrics into a dashboard someone actually glances at, and alerts that page someone before a customer complaint is the first signal anything's wrong.

1. Local Setup

# docker-compose.yml — Prometheus + Grafana, pointed at your app
services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml"]
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    depends_on: [prometheus]

# prometheus.yml
scrape_configs:
  - job_name: acme-support-agent
    metrics_path: /actuator/prometheus
    scrape_interval: 15s
    static_configs:
      - targets: ["host.docker.internal:8080"]

Add Prometheus as a Grafana data source pointing at http://prometheus:9090, and every panel below is a PromQL query away.

2. Token Throughput

sum(rate(gen_ai_client_token_usage_total{gen_ai_token_type="output"}[5m])) by (gen_ai_request_model)

rate(...[5m]) turns a cumulative counter into "tokens per second over the last 5 minutes," broken out per model — the panel that shows, at a glance, whether the support agent is still mostly running on Ollama in staging or has shifted to a hosted provider in production.

3. Latency Percentiles

Micrometer's Timer doesn't expose real percentiles to Prometheus by default — gen_ai_client_operation_seconds_max gives you the worst case, not p99. Turn on histogram buckets for this specific metric to get real quantiles:

# application.yml
management:
  metrics:
    distribution:
      percentiles-histogram:
        gen_ai.client.operation.duration: true

histogram_quantile(0.99,
  sum(rate(gen_ai_client_operation_seconds_bucket[5m])) by (le, gen_ai_request_model)
)

4. Error Rate

Spring AI's gen_ai metrics track successful call duration and token usage; they don't ship a ready-made error-rate dimension. Track it the same way this phase's Micrometer guide added support.escalations — a custom counter around the call:

try {
    return chatClient.prompt().user(message).call().content();
} catch (Exception e) {
    aiCallErrors.increment(); // Counter, registered once in the constructor
    throw e;
}

sum(rate(ai_call_errors_total[5m])) / sum(rate(gen_ai_client_operation_seconds_count[5m]))

Errors over total calls, as a ratio — the panel that answers "is the provider having a bad day" before enough customers complain to make it obvious.

5. Active Streams and Cost Estimate

Straight from the custom gauge this phase's Micrometer guide already registered:

support_streams_active

Cost is always an estimate — Grafana doesn't know your contract's actual per-token pricing, so encode it as a constant in the query:

(sum(rate(gen_ai_client_token_usage_total{gen_ai_token_type="input"}[1h])) * 0.00015
 + sum(rate(gen_ai_client_token_usage_total{gen_ai_token_type="output"}[1h])) * 0.0006) * 3600

That's (input tokens/sec × price-per-token + output tokens/sec × price-per-token) × 3600, giving an estimated dollars-per-hour panel — useful for spotting a spike, not a substitute for your provider's actual billing dashboard, which remains the source of truth.

6. Alerts

Three alerts cover most of what actually pages someone, expressed as Grafana alert rule conditions:

# Grafana alert rule (simplified)
- alert: SupportAgentLatencyP99High
  expr: histogram_quantile(0.99, sum(rate(gen_ai_client_operation_seconds_bucket[5m])) by (le)) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Support agent p99 latency above 5s"

7. Correlating With Logs

When an alert fires, the dashboard tells you that something's wrong; a log panel filtered by trace ID tells you what. Add a Loki (or equivalent) log panel to the same dashboard, filtered by the trace IDs the previous guide's tracing setup already attaches to every request — click a slow point on the latency panel, jump straight to the exact request's full log trail, including which advisor or tool call in the chain was actually slow, without grepping.

What's Next

Dashboards and alerts are watching. The next guide turns to the other side of production readiness: actually controlling what those dashboards are measuring — prompt caching, guardrails, and a model-router that keeps cost from growing unbounded in the first place.