Resilience Patterns: Timeouts, Retries & Circuit Breakers
A provider outage during peak volume takes down more than the support bot — it exhausts the thread pool and drags down the whole app. Timeouts, Resilience4j, and a fallback that actually keeps the app up.
The Outage That Took Down the Whole App
End of month, refund questions spike, and the hosted provider starts rate-limiting Acme Fintech's account. Every ChatClient call this roadmap has written so far just... waits. No timeout was ever configured, so requests that would normally take a second or two now hang for a minute or more. Within a few minutes, every thread in the application's request-handling pool is stuck waiting on a slow or failing model call — including the ones that would have served the app's unrelated health-check endpoint, or a completely different feature with no AI involvement at all. One provider having a bad day just took the whole application down.
This is the failure mode a normal REST dependency taught every backend engineer to guard against years ago — timeouts, retries, circuit breakers. AI calls need the exact same discipline, for the same reasons, just with AI-specific nuance about what's actually safe to retry.
1. Bound Any Single Call: HTTP Client Timeouts
The first, cheapest fix has nothing to do with Resilience4j — it's making sure a single call can't hang forever in the first place. Spring AI doesn't expose this as an application.yml property; each provider module ships a customizer for its underlying HTTP client:
@Bean
public OpenAiHttpClientBuilderCustomizer httpClientCustomizer() {
return builder -> builder
.connectionTimeout(Duration.ofSeconds(5))
.readTimeout(Duration.ofSeconds(30));
}This bean applies uniformly to every OpenAI-backed model in the application — chat, embedding, image, audio — not just one ChatClient. Other provider starters (Anthropic, Azure) ship an equivalent customizer interface for their own client; the pattern generalizes even though the exact interface name changes per provider.
Set this before anything else in this guide. A circuit breaker and a retry policy are both useless if the calls they're wrapping never time out in the first place — everything downstream in this guide assumes a bounded call, not an unbounded one.
2. Resilience4j: Circuit Breaker and Retry
// build.gradle
implementation 'io.github.resilience4j:resilience4j-spring-boot4'
implementation 'org.springframework.boot:spring-boot-starter-aop'@Service
public class SupportChatService {
private final ChatClient chatClient;
@CircuitBreaker(name = "supportChat", fallbackMethod = "chatFallback")
@Retry(name = "supportChat")
public String reply(String message) {
return chatClient.prompt().user(message).call().content();
}
private String chatFallback(String message, Exception e) {
log.warn("Support chat call failed, falling back", e);
return "I'm having trouble reaching our AI assistant right now — a human agent will follow up shortly.";
}
}# application.yml
resilience4j:
circuitbreaker:
instances:
supportChat:
sliding-window-size: 20
failure-rate-threshold: 50
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 5
retry:
instances:
supportChat:
max-attempts: 2
wait-duration: 500ms
retry-exceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
ignore-exceptions:
- org.springframework.ai.retry.NonTransientAiExceptionTwo nuances that matter specifically for AI calls, not generic REST resilience:
Retry only what's actually transient. A network timeout or a 5xx from the provider is worth retrying. A 400 because your prompt violated the provider's content policy, or a malformed request, will fail identically on every retry — you're just burning tokens and adding latency for a guaranteed second failure. Spring AI's NonTransientAiException hierarchy exists specifically to distinguish these; configure Resilience4j's ignore-exceptions to skip retrying them, as shown above.
@TimeLimiter needs an async return type (CompletableFuture<String>, or a reactive type) to function — it can't interrupt a synchronous, blocking .call() mid-flight. If reply() stays synchronous like the example above, the HTTP client timeout from section 1 is what actually bounds call duration; reach for @TimeLimiter only once this call is genuinely asynchronous or reactive (a streaming endpoint using .stream(), for instance).
Resilience4j's aspects apply in a fixed order — Retry(CircuitBreaker(RateLimiter(TimeLimiter(Bulkhead(call))))) — worth knowing when combining several, since it explains why a circuit breaker sees each retry attempt as a separate call for its failure-rate calculation, not the whole retried sequence as one.
3. A Fallback Worth Having: Ollama as the Backup
chatFallback above returns a static apology. A more useful fallback for a support bot specifically: fail over to the local Ollama model from this roadmap's very first guide — same ChatClient shape, dramatically lower stakes if it's slower or slightly less capable, and no dependency on the provider that's currently down:
@Service
public class SupportChatService {
private final ChatClient primaryClient; // hosted provider
private final ChatClient fallbackClient; // Ollama, local
@CircuitBreaker(name = "supportChat", fallbackMethod = "replyWithFallbackModel")
public String reply(String message) {
return primaryClient.prompt().user(message).call().content();
}
private String replyWithFallbackModel(String message, Exception e) {
log.warn("Primary provider unavailable, falling back to local Ollama", e);
return fallbackClient.prompt().user(message).call().content();
}
}This is a direct, practical payoff of this roadmap's first-guide decision to keep Ollama configured throughout — not just a free dev environment, but a legitimate degraded-mode backup that keeps the feature working, just on a different model, instead of returning an apology while the primary provider recovers.
4. Bulkhead: Stop One Slow Dependency From Exhausting Everything
This is the specific mechanism that would have prevented this guide's opening incident — capping how many AI calls can be in flight at once, so a slow provider degrades one feature's throughput instead of exhausting the shared thread pool every other endpoint depends on:
resilience4j:
bulkhead:
instances:
supportChat:
max-concurrent-calls: 25
max-wait-duration: 0@Bulkhead(name = "supportChat", fallbackMethod = "chatFallback")
@CircuitBreaker(name = "supportChat", fallbackMethod = "chatFallback")
public String reply(String message) { ... }With this in place, a provider outage means at most 25 threads are ever stuck waiting on it — the request-handling pool serving every other endpoint, including the health check that has nothing to do with AI, stays available. The 26th concurrent support-chat request gets an immediate, fast failure (routed to the same fallback) instead of queuing behind an already-overwhelmed dependency.
What's Next
The support agent now degrades gracefully instead of taking the whole app down with it. The next two guides shift away from text entirely — vision, image generation, and audio, the multimodal capabilities behind the same ChatClient and structured-output patterns this whole roadmap has already taught.
Frequently asked questions
Should every ChatClient call in the application get the full timeout + circuit breaker + retry + bulkhead treatment?
Scale it to what's actually at stake — a customer-facing support endpoint under real traffic justifies all four layers from this guide. An internal admin tool used by three people, or a background batch job with its own retry loop already, often doesn't need the same ceremony. Apply this deliberately to what genuinely needs it, not uniformly everywhere.
Does the circuit breaker's failure-rate calculation know the difference between a timeout and a content-policy rejection?
Only if you configure it to — by default Resilience4j counts any exception as a failure toward the threshold. Pair the ignore-exceptions list from this guide's retry configuration with an equivalent recordExceptions/ignoreExceptions setting on the circuit breaker itself if you don't want non-transient failures (which will happen again immediately no matter what) tripping the breaker the same way real outages do.
Is the Ollama fallback a good idea for every feature, not just support chat?
Only where a lower-capability response is genuinely acceptable as a degraded mode — a classification or triage endpoint from this roadmap's second guide might tolerate a less capable fallback model fine; a nuanced, customer-facing explanation might not. Decide per feature, the same judgment call this roadmap's model-router guide made about matching model capability to task stakes.
How do I test that these resilience patterns actually work before a real outage proves it?
Point the primary ChatClient at a deliberately unreachable endpoint or a mock that times out, in an integration test, and assert the fallback path is what actually executes — the same way you'd test a circuit breaker around any other external dependency. Don't wait for a real provider outage to be the first time this code path runs.