Cost Tracking, Prompt Caching & Guardrails

The Alert That Actually Fired

The token-budget alert from the last guide isn't hypothetical anymore — it fired last Tuesday. Daily spend on the support agent is up 40% week over week, and nobody added a new feature. This guide is the response: where the tokens are actually going, what cuts the number without cutting quality, and what stops it from happening again unnoticed.

1. Cost Tracking Per User, Per Feature

The dashboard from the last guide shows aggregate spend. Finding what's driving an increase needs it broken down — the same custom-counter pattern from this phase's Micrometer guide, tagged for cost attribution specifically:

@Service
public class SupportChatService {
 
    private final ChatClient chatClient;
    private final MeterRegistry registry;
 
    public String reply(String customerId, String feature, String message) {
        ChatResponse response = chatClient.prompt()
            .user(message)
            .call()
            .chatResponse();
 
        Usage usage = response.getMetadata().getUsage();
        registry.counter("ai.tokens.input", "feature", feature, "customer_tier", tierOf(customerId))
            .increment(usage.getPromptTokens());
        registry.counter("ai.tokens.output", "feature", feature, "customer_tier", tierOf(customerId))
            .increment(usage.getCompletionTokens());
 
        return response.getResult().getOutput().getText();
    }
}

ChatResponse.getMetadata().getUsage() — the same object Spring AI already populates on every call — gives you exact per-call token counts. Tag by feature (triage vs conversational reply vs RAG-grounded policy answer) and customer_tier, not just model, and "spend is up 40%" becomes "the new RAG-grounded policy-answer feature is 60% of spend" in one query, instead of a guess.

2. Prompt Caching: The Biggest Single Lever

The support agent's system prompt — the persona, the tool descriptions, the "never guess" RAG instruction — is identical on every single call. Sending it fresh every time is pure waste. Anthropic (and Bedrock) support prompt caching: the provider caches a prefix of the prompt server-side, and a cache hit costs a fraction of a full input token.

var cacheOptions = AnthropicCacheOptions.builder()
    .strategy(AnthropicCacheStrategy.SYSTEM_AND_TOOLS)
    .build();
 
var options = AnthropicChatOptions.builder()
    .model("claude-sonnet-4-5")
    .cacheOptions(cacheOptions)
    .build();
 
ChatResponse response = chatModel.call(new Prompt(messages, options));

SYSTEM_AND_TOOLS is the highest-value default for most of this roadmap's SupportChatService — the system prompt and tool descriptions are large, static, and repeated on literally every call. CONVERSATION_HISTORY matters specifically once MessageWindowChatMemory's 20-message window (this roadmap's second phase) means later turns in a long conversation are resending most of the earlier ones — caching that growing prefix means each new turn only pays full price for what's actually new.

3. Budgeting for Reasoning Models

This roadmap's first guide introduced reasoning ("thinking") models as spending extra hidden tokens on a reasoning chain before answering. That's real, uncached cost that doesn't show up as visible response text — a reasoning model can cost several times a standard model's price for what looks like the same short answer. Reserve reasoning models for genuinely complex judgment calls (Embabel's .withLlm(...) per-action model selection from this roadmap's fifth phase is exactly the mechanism for scoping this), not as the default for every call.

4. Guardrails

Three layers, from simplest to most involved:

// 1. Cap output length per request
ChatOptions.builder().maxTokens(500).build();

// 2. Rate limit per customer with Bucket4j
Bucket bucket = Bucket.builder()
    .addLimit(Bandwidth.classic(20, Refill.intervally(20, Duration.ofMinutes(1))))
    .build();
 
if (!bucket.tryConsume(1)) {
    throw new RateLimitExceededException("Too many requests — please wait a moment");
}

// 3. Daily token budget per customer, Redis sliding window
public boolean withinDailyBudget(String customerId, int tokensRequested) {
    String key = "token-budget:" + customerId + ":" + LocalDate.now();
    Long used = redisTemplate.opsForValue().increment(key, tokensRequested);
    redisTemplate.expire(key, Duration.ofHours(25));
    return used <= DAILY_TOKEN_BUDGET;
}

maxTokens bounds a single response; the rate limiter stops one customer from hammering the endpoint; the daily budget stops a single very long conversation (or a bug causing a retry storm) from running up an unbounded bill for one account. Layer all three — they guard against different failure modes, not the same one.

5. A Cost-Aware Model Router

The cheapest fix for "cost is too high" is often not sending every request to the same model in the first place:

@Service
public class ModelRouter {
 
    private final ChatClient cheapClient;   // e.g. gpt-5-mini, or Ollama locally
    private final ChatClient capableClient; // e.g. claude-sonnet-4-5
 
    public ChatClient routeFor(TicketCategory category, boolean isFollowUp) {
        boolean simple = category == TicketCategory.GENERAL && !isFollowUp;
        return simple ? cheapClient : capableClient;
    }
}

This is the same idea this roadmap has now applied at three different layers: ChatOptions.builder().temperature(0.0) for a classification call versus a creative one (second guide), Embabel's .withLlm(...) picking a model per @Action (fifth phase), and now an explicit router choosing between two whole ChatClient beans by task. Ticket triage — one word out, low stakes — never needed the same model as a nuanced refund-denial explanation a customer will actually read.

What's Next

This is the last guide before the roadmap's true finale. You now have every piece: a working agent across three JVM frameworks, RAG, tools, MCP, evaluation, streaming, observability, and cost control. The final guide closes the loop this whole roadmap opened — how to keep all of it current as Spring AI, Embabel, and LangChain4j keep shipping.