Resilience and Production Safety for Backend Services

Resilience and Production Safety

Most outages are not caused by exotic bugs — they're caused by ordinary operations happening at the wrong moment: a retry hitting a non-idempotent endpoint twice, a rolling deploy killing pods mid-request, a connection pool sized by guesswork instead of math, a thread pool that queues forever instead of failing fast. This guide covers the production-safety patterns that separate services that degrade gracefully under stress from services that cascade into full outages: idempotency, graceful shutdown, feature flags, and pool sizing.

1. Idempotency: Making Retries Safe

A retry is only safe if the operation it repeats is idempotent — applying it twice produces the same end state as applying it once. Networks fail in the worst possible way for this: a client can time out on the response even though the server successfully processed the request, and then legitimately retry, hitting a service that already did the work.

Implementing idempotency keys

@RestController
@RequestMapping("/payments")
public class PaymentController {
 
    private final IdempotencyStore idempotencyStore;
    private final PaymentService paymentService;
 
    @PostMapping
    public ResponseEntity<PaymentResult> charge(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody PaymentRequest request) {
 
        Optional<PaymentResult> existing = idempotencyStore.find(idempotencyKey);
        if (existing.isPresent()) {
            return ResponseEntity.ok(existing.get()); // replay, no side effect
        }
 
        // Acquire a short-lived lock on the key to prevent concurrent duplicate processing
        boolean acquired = idempotencyStore.tryLock(idempotencyKey, Duration.ofSeconds(30));
        if (!acquired) {
            throw new ConflictException("Request with this key is already being processed");
        }
 
        try {
            PaymentResult result = paymentService.charge(request);
            idempotencyStore.save(idempotencyKey, result, Duration.ofHours(24));
            return ResponseEntity.ok(result);
        } finally {
            idempotencyStore.unlock(idempotencyKey);
        }
    }
}

CREATE TABLE idempotency_keys (
    idempotency_key   VARCHAR(64) PRIMARY KEY,
    request_hash      VARCHAR(64) NOT NULL,
    response_body     JSONB NOT NULL,
    status_code       INT NOT NULL,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at        TIMESTAMPTZ NOT NULL
);

Which operations need idempotency keys, and which are naturally idempotent

2. Graceful Shutdown

When a pod is terminated — during a rolling deploy, an autoscale-down, or a node drain — Kubernetes sends SIGTERM, waits up to terminationGracePeriodSeconds, then sends SIGKILL if the process hasn't exited. What happens in that window determines whether in-flight requests complete cleanly or get dropped mid-response.

Enabling graceful shutdown in Spring Boot

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=25s

With server.shutdown=graceful, the embedded web server (Tomcat/Netty) stops accepting new connections on SIGTERM but allows in-flight requests up to the configured timeout to complete before the context closes.

# Deployment manifest
spec:
  terminationGracePeriodSeconds: 35
  containers:
    - name: order-service
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 5"]

Draining, not just stopping

Graceful HTTP shutdown alone isn't enough if you also consume from a queue or hold long-lived connections. Each resource needs to drain in the right order:

@Component
public class GracefulShutdownCoordinator {
 
    private final KafkaListenerEndpointRegistry kafkaRegistry;
    private final ApplicationEventPublisher events;
 
    @EventListener(ContextClosedEvent.class)
    public void onShutdown() {
        // 1. Stop pulling new work from the queue
        kafkaRegistry.getListenerContainers().forEach(MessageListenerContainer::stop);
 
        // 2. Flip readiness so the LB/mesh stops routing (if not already draining)
        events.publishEvent(new AvailabilityChangeEvent<>(this, ReadinessState.REFUSING_TRAFFIC));
 
        // 3. Let in-flight message processing finish (bounded by shutdown-phase timeout)
        // 4. HikariCP pool close happens automatically via Spring's bean destruction order
    }
}

3. Feature Flags for Production Safety

Feature flags decouple deployment from release — you can ship code to production dark, then turn it on for 1% of traffic, then 10%, then 100%, without a redeploy at each step. This is one of the highest-leverage tools for reducing blast radius.

@Service
public class CheckoutService {
 
    private final FeatureFlagClient flags;
 
    public Receipt checkout(Cart cart, String userId) {
        if (flags.isEnabled("new-pricing-engine", userId)) {
            return checkoutWithNewPricingEngine(cart);
        }
        return checkoutWithLegacyPricing(cart);
    }
}

public interface FeatureFlagClient {
    boolean isEnabled(String flagKey, String targetingKey);
}
 
// A minimal in-house implementation backed by a config source you can hot-reload
@Component
public class ConfigDrivenFeatureFlagClient implements FeatureFlagClient {
 
    private final AtomicReference<Map<String, FlagRule>> rules = new AtomicReference<>(Map.of());
 
    @Override
    public boolean isEnabled(String flagKey, String targetingKey) {
        FlagRule rule = rules.get().get(flagKey);
        if (rule == null) return false; // undefined flags default OFF, never ON
        return rule.matches(targetingKey);
    }
 
    @Scheduled(fixedDelay = 30_000)
    void refreshFromConfigSource() {
        rules.set(configSource.fetchLatestRules());
    }
}

4. Connection Pool Sizing with HikariCP

HikariCP is Spring Boot's default JDBC connection pool. Undersizing it causes requests to queue for a connection under load; oversizing it wastes database-side resources and, past a certain point, actually reduces throughput because the database spends more time context-switching between connections than doing work.

spring.datasource.hikari.maximum-pool-size=20
spring.datasource.hikari.minimum-idle=20
spring.datasource.hikari.connection-timeout=3000
spring.datasource.hikari.idle-timeout=600000
spring.datasource.hikari.max-lifetime=1800000
spring.datasource.hikari.leak-detection-threshold=60000

The sizing formula

HikariCP's own guidance, based on PostgreSQL's connection-pool sizing research, is a widely-cited starting formula:

pool_size = Tn × (Cm - 1) + 1

Tn = number of threads that can concurrently execute queries
Cm = number of concurrent connections/queries each thread ever holds (usually 1)

For a typical web service, this reduces to something close to: pool size ≈ core_count × 2 + effective_spindle_count, but the practical version most teams use is empirical: start with a modest pool (e.g., 10-20), watch hikaricp.connections.pending, and increase only if you observe real queueing under real load — not by guessing "bigger is safer."

Sizing across replicas, not per pod

# leak-detection-threshold in action — logs this if a connection isn't returned in 60s
# com.zaxxer.hikari.pool.ProxyLeakTask : Connection leak detection triggered for
# com.zaxxer.hikari.pool.HikariProxyConnection@... on thread http-nio-8080-exec-7,
# stack trace follows

5. Thread Pool Tuning

Every async boundary in a Spring Boot service — @Async methods, ExecutorService-backed work, Tomcat's request-handling threads — is backed by a thread pool with its own sizing trade-offs. The same principle from connection pools applies: unbounded pools and unbounded queues turn overload into an OOM instead of a controlled, fast failure.

@Configuration
@EnableAsync
public class AsyncConfig {
 
    @Bean(name = "notificationExecutor")
    public Executor notificationExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(8);
        executor.setMaxPoolSize(16);
        executor.setQueueCapacity(200);       // bounded — not Integer.MAX_VALUE
        executor.setThreadNamePrefix("notify-");
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        executor.initialize();
        return executor;
    }
}

Choosing pool size for I/O-bound vs CPU-bound work

# Tomcat's own request-handling thread pool — the other pool that matters
server:
  tomcat:
    threads:
      max: 200
      min-spare: 20
    accept-count: 100          # OS-level backlog once max threads are busy
    max-connections: 10000

Key takeaways

• Idempotency keys make retries safe for inherently non-idempotent operations (charges, order creation) — store the request hash alongside the cached response so reused keys with different payloads fail loudly instead of returning the wrong result.
• server.shutdown=graceful alone is not enough — pair it with a preStop sleep to cover the race between SIGTERM delivery and Service endpoint propagation.
• Drain every stateful resource in order on shutdown: stop accepting new work first, let in-flight work finish, then close pools and consumers last.
• Feature flags decouple deploy from release and turn incident mitigation into a config change instead of a rollback — but treat them as temporary and audit for dead flags regularly.
• Bigger connection pools are not automatically better throughput — size against actual database CPU parallelism across all replicas combined, not per-pod guesswork.
• Always bound thread pool queues and pick a deliberate rejection policy; unbounded queues convert overload into a slow-motion OOM instead of a fast, visible failure.
• Watch hikaricp.connections.pending and thread-pool queue depth as leading indicators of saturation before they become customer-facing latency.

Interview Questions

• What makes an operation idempotent, and why do retries require it?
• How would you design an idempotency-key mechanism for a payment API? What happens if the same key is reused with a different payload?
• Walk through what happens between a SIGTERM being sent to a pod and the pod actually being removed from traffic in Kubernetes. Where's the race condition, and how do you close it?
• Why is a preStop hook with a short sleep often necessary even with graceful shutdown enabled?
• In what order should a service drain its resources during shutdown — HTTP server, message consumers, DB pool — and why does the order matter?
• What's the difference between a release flag and a kill switch? Why might a team keep kill switches around permanently?
• What risk does an unbounded feature flag inventory create over time?
• Explain HikariCP's connection pool sizing formula. Why doesn't "bigger pool = more throughput" hold indefinitely?
• If hikaricp.connections.pending is climbing under load, what are the possible root causes, and how would you distinguish between them?
• Why is Executors.newCachedThreadPool() risky in a production service handling variable load?
• What's the difference between CPU-bound and I/O-bound thread pool sizing, and why does that distinction matter?
• What does leak-detection-threshold in HikariCP help you catch, and what would you do after seeing that warning in logs?
• Describe a RejectedExecutionHandler strategy and when you'd choose CallerRunsPolicy versus failing the request outright.
• How do connection pool size, replica count, and database max_connections interact in a horizontally scaled deployment?