Resilience Patterns: Retries, Circuit Breakers, Bulkheads

Resilience Patterns

Every network call in a distributed system can fail in ways a local function call never does: slow, partially failed, or entirely unreachable. Without deliberate resilience patterns, a single struggling downstream service can consume every thread in every upstream caller within seconds — turning one slow dependency into a full outage. This guide covers the core patterns — retries, timeouts, fallbacks, circuit breakers, and bulkheads — using Resilience4j on Spring Boot, and how they compose into graceful degradation instead of cascading failure.

1. Why Cascading Failure Happens

The mechanism is almost always the same: a downstream service degrades, upstream callers keep sending requests and waiting, their threads (or connections) pile up, and the upstream service itself becomes unavailable — even though nothing is wrong with its own code.

2. The Resilience4j Toolbox

Resilience4j is a lightweight, composable resilience library for Java, and the standard choice in the Spring Boot ecosystem since Hystrix entered maintenance mode. Each pattern is a separate, independently configurable module that can be layered together.

<!-- pom.xml -->
<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.2.0</version>
</dependency>

3. Retries: Handling Transient Failures

Retries assume a failure might be transient — a brief network blip, a momentary GC pause on the callee — and that trying again shortly after has a real chance of succeeding.

@Service
public class InventoryClientService {
 
    private final InventoryClient inventoryClient;
 
    @Retry(name = "inventoryService", fallbackMethod = "fallbackAvailability")
    public InventoryAvailability checkAvailability(String sku) {
        return inventoryClient.checkAvailability(sku);
    }
 
    private InventoryAvailability fallbackAvailability(String sku, Exception ex) {
        // Fail-safe default when retries are exhausted
        return InventoryAvailability.unknown(sku);
    }
}

resilience4j:
  retry:
    instances:
      inventoryService:
        max-attempts: 3
        wait-duration: 200ms
        # Exponential backoff avoids synchronized retry storms
        exponential-backoff-multiplier: 2
        retry-exceptions:
          - java.io.IOException
          - org.springframework.web.client.ResourceAccessException
        ignore-exceptions:
          - com.example.SkuNotFoundException   # 404s are not transient — don't retry

@Bean
public RetryConfig retryConfig() {
    return RetryConfig.custom()
        .maxAttempts(3)
        .intervalFunction(
            IntervalFunction.ofExponentialRandomBackoff(
                Duration.ofMillis(200),   // initial interval
                2.0,                       // multiplier
                0.5                        // randomization factor (jitter)
            ))
        .retryExceptions(IOException.class, ResourceAccessException.class)
        .build();
}

4. Timeouts and Time Limiters

A Time Limiter enforces a maximum duration for an asynchronous call, converting a hang into a fast, well-defined failure the rest of the system can react to.

@TimeLimiter(name = "inventoryService")
@CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackAvailability")
public CompletableFuture<InventoryAvailability> checkAvailabilityAsync(String sku) {
    return CompletableFuture.supplyAsync(() -> inventoryClient.checkAvailability(sku));
}
 
private CompletableFuture<InventoryAvailability> fallbackAvailability(String sku, Throwable ex) {
    return CompletableFuture.completedFuture(InventoryAvailability.unknown(sku));
}

resilience4j:
  timelimiter:
    instances:
      inventoryService:
        timeout-duration: 2s
        cancel-running-future: true

5. Circuit Breakers: Failing Fast Instead of Piling Up

A circuit breaker tracks the recent success/failure rate of calls to a dependency and, once failures cross a threshold, stops calling it entirely for a cooldown period — failing immediately instead of waiting for timeouts on every request. This protects both the caller (fast failure, no thread pileup) and the struggling callee (no added load while it recovers).

@Service
public class InventoryClientService {
 
    @CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackAvailability")
    @Retry(name = "inventoryService")
    public InventoryAvailability checkAvailability(String sku) {
        return inventoryClient.checkAvailability(sku);
    }
 
    private InventoryAvailability fallbackAvailability(String sku, Throwable ex) {
        if (ex instanceof CallNotPermittedException) {
            // Circuit is OPEN — we didn't even attempt the call
            log.warn("Circuit open for inventory-service, using cached availability");
        }
        return cache.getLastKnownAvailability(sku)
            .orElse(InventoryAvailability.unknown(sku));
    }
}

resilience4j:
  circuitbreaker:
    instances:
      inventoryService:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        slow-call-rate-threshold: 80
        slow-call-duration-threshold: 2s
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 5
        automatic-transition-from-open-to-half-open-enabled: true
        record-exceptions:
          - java.io.IOException
          - org.springframework.web.client.ResourceAccessException
        ignore-exceptions:
          - com.example.SkuNotFoundException

Circuit breaker configuration decisions

6. Fallbacks: Degrading Gracefully

A fallback is what happens when the primary path fails and retries/circuit breaker have given up. The goal is graceful degradation, not silent failure — the user gets something useful, or at least a clear, fast error, instead of a hung request.

7. Bulkheads: Isolating Failure Domains

A bulkhead (named after the watertight compartments in a ship's hull) limits how many concurrent calls can be in flight to a given dependency, so that one slow or failing dependency cannot consume all of a service's threads or connections — leaving capacity for calls to other, healthy dependencies.

Resilience4j offers two bulkhead implementations:

resilience4j:
  # Semaphore bulkhead — caps concurrent calls with a permit count
  # Good default: low overhead, works with blocking (Feign/MVC) code
  bulkhead:
    instances:
      inventoryService:
        max-concurrent-calls: 20
        max-wait-duration: 10ms
 
  # Thread pool bulkhead — runs calls on a dedicated, bounded executor
  # Provides true isolation (separate threads) at the cost of a context switch
  thread-pool-bulkhead:
    instances:
      reportingService:
        max-thread-pool-size: 10
        core-thread-pool-size: 5
        queue-capacity: 20

@Bulkhead(name = "inventoryService", type = Bulkhead.Type.SEMAPHORE)
@CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackAvailability")
public InventoryAvailability checkAvailability(String sku) {
    return inventoryClient.checkAvailability(sku);
}
 
@Bulkhead(name = "reportingService", type = Bulkhead.Type.THREADPOOL)
public CompletableFuture<Report> generateReport(String reportId) {
    return CompletableFuture.supplyAsync(() -> reportingClient.fetch(reportId));
}

8. Composing the Patterns: How They Work Together

None of these patterns work well in isolation — a circuit breaker without a bulkhead can still let a slow dependency exhaust threads while the breaker is still CLOSED and gathering failure data; a retry without backoff can amplify an outage; a bulkhead without a fallback just fails fast without recovering gracefully.

Recommended composition order

1. Bulkhead (outermost) — reject immediately if we're already at max concurrency for this dependency; don't even attempt.
2. Circuit Breaker — if the dependency is known unhealthy, fail fast without a network call.
3. Time Limiter — bound how long any single attempt (including retries) is allowed to take.
4. Retry (innermost, closest to the actual call) — absorb transient blips with backoff and jitter.
5. Fallback — attached at the outermost level, invoked whenever any layer above gives up.

@Bulkhead(name = "inventoryService")
@CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackAvailability")
@Retry(name = "inventoryService")
public InventoryAvailability checkAvailability(String sku) {
    return inventoryClient.checkAvailability(sku);
}

9. Testing Resilience: Don't Just Configure It, Verify It

Resilience configuration that has never been exercised under real failure conditions is a hope, not a guarantee. Two complementary approaches:

• Unit/integration tests: Resilience4j ships an in-memory CircuitBreakerRegistry you can assert against directly — force a dependency to fail N times and verify the circuit opens, the fallback fires, and it recovers after the wait duration.
• Chaos testing: tools like Chaos Monkey for Spring Boot (or manual fault injection — killing a pod, adding artificial latency with a service mesh fault injection rule) validate the actual runtime behavior under failure, including interactions between multiple patterns that are hard to reason about statically.

@Test
void circuitBreakerOpensAfterFailureThreshold() {
    CircuitBreaker circuitBreaker = circuitBreakerRegistry.circuitBreaker("inventoryService");
 
    // Simulate failures crossing the configured threshold
    for (int i = 0; i < 10; i++) {
        circuitBreaker.onError(0, TimeUnit.MILLISECONDS, new IOException("simulated"));
    }
 
    assertThat(circuitBreaker.getState()).isEqualTo(CircuitBreaker.State.OPEN);
}

Key takeaways

• Cascading failure is usually caused by slow (not hard) dependency failures holding caller threads open — timeouts are the first and most important defense.
• Only retry idempotent operations, and always use exponential backoff with jitter to avoid synchronized retry storms against a recovering service.
• Circuit breakers fail fast once a failure/slow-call threshold is crossed, protecting both the caller's thread pool and the struggling dependency from further load.
• Order matters: bulkhead and circuit breaker should wrap retry, not the reverse, so retries are measured as part of a single logical call rather than each counted separately.
• Not every dependency deserves a soft fallback — for strictly consistency-sensitive operations (payments, final inventory decrement), fail clearly rather than fabricate a plausible success.
• Bulkheads isolate failure domains so one degraded dependency can't starve capacity needed for calls to unrelated, healthy dependencies — including at the HTTP connection pool level, not just the thread level.
• These patterns compose: bulkhead → circuit breaker → time limiter → retry → fallback is the standard layering, each catching what the previous layer didn't.
• Monitor circuit breaker state transitions and bulkhead rejection rates directly — they surface unhealthy dependencies earlier than aggregate error-rate dashboards, and validate the whole stack with chaos testing or a deliberate game day before trusting it in production.

Interview Questions

• Why is a slow dependency more dangerous than a hard-down dependency in a distributed system?
• Walk through the three states of a Resilience4j circuit breaker and what triggers each transition.
• Why should retries use exponential backoff with jitter instead of a fixed delay?
• When is it unsafe to retry a failed request, and how would you make a non-idempotent operation safely retryable?
• What is the difference between a semaphore bulkhead and a thread pool bulkhead in Resilience4j?
• Why should @CircuitBreaker typically wrap @Retry rather than the other way around?
• Give an example of a dependency where a fallback should NOT silently degrade, and explain why.
• How does a bulkhead prevent one slow dependency from affecting calls to a completely different, healthy dependency?
• Why is a shared HTTP connection pool across multiple downstream services a resilience risk, even with a circuit breaker configured?
• What metrics would you monitor to detect an unhealthy dependency before it shows up in overall error rate dashboards?
• How would you test that a circuit breaker actually opens and recovers as configured, without waiting for a real production incident?
• What's the purpose of the HALF_OPEN state in a circuit breaker, and why not go straight from OPEN back to CLOSED?
• How do timeouts, retries, circuit breakers, and bulkheads compose together to prevent cascading failure? Describe the recommended layering order.
• What's the risk of setting wait-duration-in-open-state too short? Too long?