Micrometer Observability: Answering 'How Many Tokens Did We Spend?'

The Question From Guide One, Finally Answered

Back in this roadmap's very first guide, the fintech team's opening frustration with hand-rolled provider calls included this line: "Nobody can answer 'how many tokens did we spend on this feature last week' without grepping logs." Seven phases later, the support agent calls a model for triage, RAG, tool use, and Embabel-planned refund adjudication — more surface area for that same question, not less. This guide is where it finally gets answered properly, with metrics instead of grep.

1. It's Already Instrumenting — You Just Haven't Looked

Every ChatModel, EmbeddingModel, and VectorStore call Spring AI makes is wrapped in a Micrometer Observation automatically, the moment Micrometer's on the classpath — no code change anywhere in the six phases of ChatClient calls this roadmap has already written.

// build.gradle
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'io.micrometer:micrometer-registry-prometheus'

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: prometheus, health
  metrics:
    tags:
      application: acme-support-agent

That's the entire setup. /actuator/prometheus is now a scrapeable endpoint with every AI call this application makes already showing up in it.

2. The Metrics That Actually Exist

Two Micrometer instruments matter most, under their logical names (what you'd search for in code or a Micrometer-aware backend) and their Prometheus-exported form (what you'll actually see scraping /actuator/prometheus):

# What a scrape actually looks like
gen_ai_client_operation_seconds_count{gen_ai_operation_name="chat", gen_ai_request_model="gpt-5-mini"} 1847
gen_ai_client_operation_seconds_sum{gen_ai_operation_name="chat", gen_ai_request_model="gpt-5-mini"} 412.3
gen_ai_client_token_usage_total{gen_ai_token_type="input", gen_ai_request_model="gpt-5-mini"} 891204
gen_ai_client_token_usage_total{gen_ai_token_type="output", gen_ai_request_model="gpt-5-mini"} 203417

That's the direct answer to guide one's unanswered question: sum(gen_ai_client_token_usage_total{gen_ai_token_type="output"}) over the last seven days, in whatever query language your metrics backend speaks. No more grepping logs for it.

3. Prompt and Completion Content: Opt In Deliberately

Duration and token counts are on by default. The actual prompt and response text are not, for good reason:

spring:
  ai:
    chat:
      observations:
        log-prompt: true
        log-completion: true

Enable them narrowly — a feature flag scoped to a specific investigation, or a non-production environment — rather than leaving them on in production by default.

4. Custom Metrics: Extending Beyond Auto-Instrumentation

Auto-instrumentation covers the model call itself. It doesn't know about your business logic — track that with ordinary Micrometer, the same way you'd instrument any other Spring service:

@Service
public class SupportChatService {
 
    private final ChatClient chatClient;
    private final Counter escalationCounter;
    private final AtomicInteger activeStreams;
 
    public SupportChatService(ChatClient.Builder builder, ChatMemory chatMemory, MeterRegistry registry) {
        this.chatClient = builder
            .defaultAdvisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
            .build();
        this.escalationCounter = Counter.builder("support.escalations")
            .description("Number of conversations escalated to a human agent")
            .register(registry);
        this.activeStreams = registry.gauge("support.streams.active", new AtomicInteger(0));
    }
 
    public Flux<String> reply(String ticketId, String message) {
        activeStreams.incrementAndGet();
        return chatClient.prompt()
            .user(message)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, ticketId))
            .stream()
            .content()
            .doFinally(signal -> activeStreams.decrementAndGet());
    }
}

support.escalations and support.streams.active now sit right alongside gen_ai_client_token_usage_total in the same /actuator/prometheus output — one dashboard, business metrics and model metrics together, exactly what the next guide builds.

5. Tracing: Following One Request Across Every Call

A single customer message can trigger several model calls in this roadmap's more advanced flows — a RAG retrieval, a tool call, an Embabel-planned multi-action agent. Micrometer's tracing integration (via Micrometer Tracing, exportable to Zipkin or an OpenTelemetry-compatible backend) gives each of those calls a shared trace ID, so "why was this one response slow" becomes a single trace to inspect instead of guessing which of four model calls was the bottleneck.

// build.gradle
implementation 'io.micrometer:micrometer-tracing-bridge-otel'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp'

With that on the classpath, every Spring AI Observation — and every log line inside the same request, once your logging pattern includes %X{traceId} — carries the same trace ID automatically. The next guide's Grafana setup uses exactly this to jump from "latency spiked" straight to "here's the trace that was slow."

What's Next

Metrics are flowing to /actuator/prometheus; nobody's looking at them yet. The next guide turns this into an actual dashboard — token throughput, latency percentiles, error rate, and the alerts that page someone before a customer complains.