Streaming Responses: Fixing the Spinner Problem

Six Seconds of Nothing

Every ChatClient call across this whole roadmap has used .call() — wait for the entire response, then return it. That's fine for a triage endpoint returning one word. It's a genuinely bad experience for SupportChatService.reply(...) from this roadmap's second phase, drafting a multi-paragraph explanation: the customer sends a message and stares at a loading spinner for four to six seconds while the model generates the whole thing before a single character reaches them.

The fix isn't making the model faster — it's not waiting for it to finish before showing anything. Streaming sends tokens to the browser as the model generates them, the same way ChatGPT's UI visibly "types" a response instead of pasting it all at once. The total time to the last token doesn't change; the time to the first visible word drops from seconds to a couple hundred milliseconds, which is what "feels fast" actually means to a user.

1. Three Ways to Stream

.stream() replaces .call() on the same ChatClient chain — everything else about the call (advisors, tools, system prompt) stays identical:

Flux<String> tokens = chatClient.prompt()
    .user("Draft a reply explaining why order 4521's refund was denied.")
    .stream()
    .content();

Three terminal methods, each returning progressively more detail as a Flux:

Flux<String> content = chatClient.prompt().user(msg).stream().content();
// Just the text, token by token — what you want 90% of the time.
 
Flux<ChatResponse> responses = chatClient.prompt().user(msg).stream().chatResponse();
// Each chunk wrapped with metadata — token usage, finish reason, model info.
 
Flux<ChatClientResponse> clientResponses = chatClient.prompt().user(msg).stream().chatClientResponse();
// Also exposes advisor execution context — e.g. the documents QuestionAnswerAdvisor
// retrieved for this call, useful if the UI wants to show citations as they resolve.

2. Exposing It as Server-Sent Events

Flux<String> returned from a Spring MVC or WebFlux controller, with the right content type, is a Server-Sent Events stream — no separate SSE library needed:

@RestController
@RequestMapping("/api/support")
public class SupportStreamController {
 
    private final ChatClient chatClient;
 
    public SupportStreamController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }
 
    @PostMapping(value = "/{ticketId}/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> stream(@PathVariable String ticketId, @RequestBody String message) {
        return chatClient.prompt()
            .user(message)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, ticketId))
            .stream()
            .content();
    }
}

produces = MediaType.TEXT_EVENT_STREAM_VALUE is the entire integration point — Spring's reactive web stack takes care of framing each emitted String as an SSE data: event and keeping the HTTP connection open until the Flux completes.

3. Consuming It in the Browser

The native EventSource API is built for exactly this — no extra frontend dependency for a basic case:

const eventSource = new EventSource(`/api/support/${ticketId}/stream`);
let fullReply = "";
 
eventSource.onmessage = (event) => {
    fullReply += event.data;
    replyElement.textContent = fullReply; // append, don't replace
};
 
eventSource.onerror = () => {
    eventSource.close();
};

4. Backpressure and Disconnection

A customer closing their laptop mid-response doesn't stop the model from generating tokens unless your code notices the connection is gone:

@PostMapping(value = "/{ticketId}/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> stream(@PathVariable String ticketId, @RequestBody String message) {
    return chatClient.prompt()
        .user(message)
        .stream()
        .content()
        .doOnCancel(() -> log.info("Client disconnected mid-stream for ticket {}", ticketId))
        .timeout(Duration.ofSeconds(30));
}

Flux cancellation propagates automatically when the client disconnects — Spring's reactive stack stops pulling tokens, and the underlying provider call is cancelled rather than continuing to burn tokens for a response nobody will see. .doOnCancel(...) is there for observability (log it, decrement an in-flight-requests gauge — this roadmap's next guide adds exactly that metric), and .timeout(...) protects against a provider that hangs without ever completing or erroring.

5. What Doesn't Compose With Streaming

Everything from this roadmap composes with .stream() except one thing:

Tools, RAG advisors, and memory advisors all work identically under .stream() — the model still calls tools and retrieves documents before it starts generating the streamed reply; only the final text generation is what streams token by token.

What's Next

Streaming fixes what the customer experiences. The next guide covers what you need to see — instrumenting every one of these calls with Micrometer so token usage, latency, and error rates are visible before a support-ticket-shaped disaster is the first time you hear about a problem.