Mini Project 10: Resilient Multi-Provider Failover Router
A circuit breaker trips on provider outage and fails over to a LangChain4j-backed alternate model, proven under a real failure, not a side-by-side sample.
Every Request Hanging Until It Times Out
A provider outage doesn't announce itself cleanly — it shows up as requests taking longer and longer, error rates creeping up, and if nothing intervenes, every in-flight request eventually hangs until its timeout fires, one at a time, while the feature depending on it is effectively down for the duration. This project builds the thing that actually prevents that: a circuit breaker that trips on sustained failure and reroutes traffic to a different provider automatically, proven under a real simulated failure — not just demonstrated as a side-by-side code sample the way this roadmap's "Spring AI and LangChain4j coexistence" discussion first introduced the idea.
Setup
// build.gradle
dependencies {
implementation 'org.springframework.ai:spring-ai-openai-spring-boot-starter' // primary
implementation 'dev.langchain4j:langchain4j-anthropic-spring-boot-starter' // failover
implementation 'org.springframework.boot:spring-boot-starter-aop' // Resilience4j needs AOP
implementation 'io.github.resilience4j:resilience4j-spring-boot3'
}# application.yml
resilience4j:
circuitbreaker:
instances:
primaryProvider:
sliding-window-size: 20
failure-rate-threshold: 50 # trip if >= 50% of the last 20 calls failed
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 5
timelimiter:
instances:
primaryProvider:
timeout-duration: 8sThis project assumes the resilience patterns (timeout, retry, circuit breaker, fallback) from this roadmap's eighth phase and the dual-framework coexistence pattern from the sixth — if @CircuitBreaker and CircuitBreakerRegistry are unfamiliar, that phase covers the mechanism this project applies specifically to a model-provider failure.
1. Primary Traffic on Spring AI, Wrapped in a Circuit Breaker
@Service
public class ResilientChatService {
private final ChatClient primaryChatClient; // Spring AI, OpenAI-backed
private final FailoverChatService failoverService;
private final CircuitBreaker circuitBreaker;
public ResilientChatService(
ChatClient.Builder primaryBuilder,
FailoverChatService failoverService,
CircuitBreakerRegistry registry
) {
this.primaryChatClient = primaryBuilder.build();
this.failoverService = failoverService;
this.circuitBreaker = registry.circuitBreaker("primaryProvider");
}
public String chat(String message) {
Supplier<String> primaryCall = () -> primaryChatClient.prompt()
.user(message)
.call()
.content();
Supplier<String> guarded = CircuitBreaker.decorateSupplier(circuitBreaker, primaryCall);
try {
return guarded.get();
} catch (CallNotPermittedException e) {
// Circuit is OPEN — primary is failing fast without even attempting the call
return failoverService.chat(message);
} catch (Exception e) {
// Circuit is still CLOSED or HALF_OPEN, but this specific call failed
return failoverService.chat(message);
}
}
}The distinction between the two catch branches matters for observability even though both currently do the same thing: CallNotPermittedException means the breaker has already decided the primary is unhealthy and didn't even attempt the call (fast, cheap failover); the generic Exception means this individual call failed while the breaker is still watching to decide. Logging them separately makes it possible to tell "the primary is confirmed down" apart from "one call had a blip" in your dashards later.
2. The Failover Path: LangChain4j, a Genuinely Different Provider
Failing over to a different model on the same provider doesn't help if the provider itself is down. The failover path is a real alternate provider, via LangChain4j — the concrete version of this roadmap's "run both frameworks in one app" lesson:
@Service
public class FailoverChatService {
private final ChatLanguageModel anthropicModel; // LangChain4j, Anthropic-backed
public String chat(String message) {
return anthropicModel.generate(message);
}
}@Configuration
public class FailoverModelConfig {
@Bean
ChatLanguageModel anthropicModel(@Value("${ANTHROPIC_API_KEY}") String apiKey) {
return AnthropicChatModel.builder()
.apiKey(apiKey)
.modelName("claude-sonnet-4-5")
.build();
}
}A failover response should be recognizably a failover response somewhere in your observability — different providers can have subtly different behavior on edge cases (formatting conventions, refusal wording), and an on-call engineer debugging "why does this response look slightly off" needs to know a failover happened at all. Tag failover responses in logs and metrics, don't let them look identical to a normal primary-path response in your telemetry.
3. Detecting Recovery: Half-Open State
Resilience4j's HALF_OPEN state (configured above: 5 trial calls after the 30-second wait-duration-in-open-state) is what stops a recovered primary provider from staying bypassed forever — it's handled automatically by the CircuitBreaker.decorateSupplier wrapper, but it's worth understanding what's actually happening:
While OPEN, every call to the primary is short-circuited straight to CallNotPermittedException without even attempting the network call — cheap, fast failover, no thundering herd against a provider that's already struggling. After the wait duration, the breaker allows a small number of trial calls through (HALF_OPEN); if those succeed, it closes and primary traffic resumes; if they still fail, it reopens and waits again.
Putting It Together
@Service
public class ResilientChatService {
private final ChatClient primaryChatClient;
private final FailoverChatService failoverService;
private final CircuitBreaker circuitBreaker;
private final MeterRegistry meterRegistry; // this roadmap's observability phase
public String chat(String message) {
Supplier<String> guarded = CircuitBreaker.decorateSupplier(
circuitBreaker,
() -> primaryChatClient.prompt().user(message).call().content()
);
try {
String response = guarded.get();
meterRegistry.counter("chat.provider", "provider", "primary").increment();
return response;
} catch (Exception e) {
meterRegistry.counter("chat.provider", "provider", "failover").increment();
return failoverService.chat(message);
}
}
}A metric tagged by which provider actually served each response turns "did failover work during that outage" from a question you'd answer by reading logs after the fact into a dashboard you can watch live — the same observability discipline this roadmap's production phase asks for applied specifically to the failover path this project exists to prove out.
What's Next
This project's circuit-breaker-plus-fallback shape is the same one that could sit in front of any external dependency, not just a model provider — worth revisiting this roadmap's resilience-patterns guide if you want the broader pattern (retry with backoff, bulkheads) beyond the circuit breaker used here.
Frequently asked questions
Why LangChain4j for the failover path instead of just configuring Spring AI with a second provider's ChatModel bean?
Spring AI supports multiple ChatModel beans and could serve as its own failover target — this project deliberately uses LangChain4j instead specifically to prove out the dual-framework coexistence pattern under a real failure, not because Spring AI can't do a same-framework failover. If your team only uses Spring AI, a second ChatModel bean pointed at a different provider is a simpler version of the same idea.
What happens to a request that's in-flight on the primary when the circuit breaker trips mid-call?
The timeout configuration (8s in this guide's TimeLimiter) bounds how long any single in-flight call can hang before it's counted as a failure — the circuit breaker's trip decision is based on the failure rate across the sliding window, not on cancelling requests already in progress. A request that was already running when the trip happened still runs to its own timeout or completion; the trip only affects new calls.
How do you test that failover actually works without waiting for a real provider outage?
Point the primary ChatClient at a URL that reliably fails (a wrong port, a deliberately misconfigured API key) in a test profile, and assert that chat() still returns a response and that it came from the failover path — the same chaos-engineering mindset as testing any other resilience pattern: force the failure deliberately rather than hoping you'll catch a bug during a real incident.
Should the failover model be a cheaper/faster model than primary, or an equivalent-quality one?
That's a product decision balancing cost against user experience during an outage — a noticeably lower-quality failover model is a worse experience than the primary being briefly unavailable with a clear "service degraded" message, depending on the use case. For most production features, an equivalent-tier model from a different provider (as this guide uses) is the safer default; a cheaper failover is reasonable if the feature is lower-stakes and cost during a rare outage matters more than parity.