Kafka Reliability and Scaling: Replication, Rebalancing, and DLQs

Kafka Reliability and Scaling

Understanding topics and partitions gets you a working pipeline. Making it survive broker failures, network partitions, buggy deploys, and traffic spikes without losing or duplicating data is a different discipline. This guide covers what actually determines durability and correctness in production Kafka: replication and the in-sync replica set, how consumer group rebalancing works and why it's often the biggest source of consumption pauses, how to reason about delivery semantics honestly, and the retry/DLQ patterns that keep a bad message from taking down your whole pipeline.

1. Replication and the In-Sync Replica Set

Every partition has a replication factor — the number of broker copies of that partition's data. One replica is the leader, handling all reads and writes; the rest are followers, continuously fetching from the leader to stay caught up.

The ISR (in-sync replica set) is the subset of replicas — leader plus followers — that are fully caught up with the leader within replica.lag.time.max.ms. A follower falling too far behind (slow disk, network issues, GC pause) is removed from the ISR until it catches back up.

acks and the durability/latency trade-off

# Producer durability configuration for a payments-critical topic
acks=all
min.insync.replicas=2
enable.idempotence=true
retries=2147483647
delivery.timeout.ms=120000

2. Consumer Group Rebalancing

Rebalancing is the process of reassigning partitions among consumers in a group — triggered when a consumer joins, leaves, crashes, or the topic's partition count changes. It is coordinated by a group coordinator broker.

Eager vs cooperative rebalancing

# Recommended: incremental cooperative rebalancing
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

What triggers rebalances in practice

• A consumer instance is deployed, scaled up, or scaled down (rolling deploys naturally cause rebalances).
• A consumer crashes or is killed without a clean shutdown, and the broker detects the missed heartbeat (session.timeout.ms).
• A consumer's poll loop takes longer than max.poll.interval.ms — the broker assumes it's dead and evicts it, even if the process is still alive, because it hasn't called poll() recently enough.
• Partitions are added to the subscribed topic.

3. Delivery Semantics

Kafka supports three delivery semantics, and the honest answer to "which one do I have" depends on both producer and consumer configuration — it isn't a single global setting.

Idempotent producer

# Prevents duplicate writes caused by producer-side retries
enable.idempotence=true

With idempotence enabled, the producer attaches a sequence number per partition; the broker deduplicates retried sends that would otherwise create duplicate records due to network retries after an ack was lost in transit. This is a prerequisite for exactly-once pipelines and cheap enough to enable by default.

At-least-once in practice (the pragmatic default)

Most production systems target at-least-once delivery with idempotent consumers — accept that duplicates can happen, and design consumers so processing a message twice has no harmful effect.

@Service
public class OrderEventConsumer {
 
    private final ProcessedEventRepository processedEvents;
    private final OrderService orderService;
 
    @KafkaListener(topics = "orders", groupId = "billing-service")
    public void onOrderEvent(OrderEvent event,
                              @Header(KafkaHeaders.RECEIVED_KEY) String key) {
        String eventId = event.eventId(); // unique ID set by the producer
 
        // Idempotency check: skip if already processed
        if (processedEvents.existsByEventId(eventId)) {
            log.info("Duplicate event {} ignored", eventId);
            return;
        }
 
        orderService.chargeCustomer(event);
        processedEvents.markProcessed(eventId);
    }
}

4. Manual Offset Acknowledgement Patterns

By default, Spring Kafka auto-commits offsets on a timer (enable.auto.commit=true), which is an at-most-once-leaning default that most production systems override for stronger guarantees.

# Disable auto-commit — take manual control
spring.kafka.consumer.enable-auto-commit=false

@Configuration
public class KafkaConsumerConfig {
 
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
            ConsumerFactory<String, OrderEvent> consumerFactory) {
        ConcurrentKafkaListenerContainerFactory<String, OrderEvent> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        // MANUAL_IMMEDIATE: commit right after the listener method returns successfully
        factory.getContainerProperties()
            .setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
        return factory;
    }
}

@KafkaListener(topics = "orders", groupId = "billing-service")
public void onOrderEvent(OrderEvent event, Acknowledgment ack) {
    try {
        orderService.chargeCustomer(event);
        ack.acknowledge();  // commit only after successful processing
    } catch (TransientException e) {
        // Don't acknowledge — message will be redelivered on next poll/rebalance
        throw e;
    }
}

Ack mode comparison

5. Retries and Dead-Letter Queues

Not every message failure should be retried forever, and not every failure should immediately fail the whole partition. The standard pattern layers in-memory retries with backoff, then routes to a retry topic or dead-letter topic (DLT) after exhausting attempts — keeping a single poison message from blocking the entire partition.

@Configuration
public class KafkaErrorHandlingConfig {
 
    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> kafkaTemplate) {
        // Publishes to <topic>.DLT after retries are exhausted
        DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(kafkaTemplate);
 
        // Exponential backoff: start 1s, double each time, cap at 30s, max 5 attempts
        ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(5);
        backOff.setInitialInterval(1_000L);
        backOff.setMultiplier(2.0);
        backOff.setMaxInterval(30_000L);
 
        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
 
        // Don't retry on deserialization or business validation errors — dead-letter immediately
        handler.addNotRetryableExceptions(
            DeserializationException.class,
            IllegalArgumentException.class
        );
        return handler;
    }
}

DLT consumer for reprocessing

@KafkaListener(topics = "orders.DLT", groupId = "billing-service-dlt")
public void onDeadLetter(
        OrderEvent event,
        @Header(KafkaHeaders.DLT_EXCEPTION_MESSAGE) String exceptionMessage,
        @Header(KafkaHeaders.DLT_ORIGINAL_TOPIC) String originalTopic) {
    log.error("Dead-lettered event {} from {}: {}",
        event.orderId(), originalTopic, exceptionMessage);
    incidentQueue.raise(event, exceptionMessage);
}

6. Scaling Considerations

Scaling a Kafka pipeline is a joint function of partition count, consumer group size, and per-message processing cost.

• Vertical scaling (concurrency within an instance): increasing concurrency in ConcurrentKafkaListenerContainerFactory adds consumer threads inside a single pod, capped by available partitions.
• Horizontal scaling (more pods): each new pod's consumers join the same group and get assigned a share of partitions, triggering a rebalance.
• Processing-time reduction: if per-message work is slow (synchronous downstream calls), consider async processing patterns, batching, or moving heavy work to a separate downstream topic/consumer rather than adding partitions purely to compensate.

Production observations

• acks=all + min.insync.replicas=2 + replication.factor=3 is the standard durable baseline; anything less accepts a defined risk of data loss on broker failure.
• Cooperative sticky rebalancing should be the default choice for any consumer group where rebalance-induced pauses matter — which is nearly all of them at scale.
• max.poll.interval.ms exceedance is the most common cause of "mystery" rebalances — always correlate rebalance events with processing latency, not just deploys and crashes.
• At-least-once + idempotent consumer is the pragmatic default for the vast majority of systems; reserve Kafka transactions (true EOS) for Kafka-to-Kafka stream processing pipelines where the complexity is justified.
• A DLT with alerting is not optional for any consumer processing business-critical events — without one, a single bad message can silently stall a partition.

Key takeaways

• Durability comes from the combination of acks=all, min.insync.replicas, and replication factor — no single setting alone guarantees it.
• Cooperative sticky rebalancing avoids stopping unaffected consumers during a rebalance; prefer it over the legacy eager assignors in new deployments.
• Slow message processing exceeding max.poll.interval.ms triggers rebalances even on healthy consumers — this is the most misdiagnosed production issue in Kafka consumption.
• Exactly-once semantics is scoped to Kafka-to-Kafka; anything writing outside Kafka still needs idempotent writes.
• Commit offsets after successful processing, not before, to get at-least-once rather than at-most-once semantics.
• Pair retries with a bounded attempt count and a dead-letter topic — unbounded blocking retries on a poison message stall the whole partition.
• Scaling consumers beyond the partition count does nothing; check whether your bottleneck is partition-bound or processing-time-bound before repartitioning.

Interview Questions

• What is the in-sync replica set (ISR), and how does it relate to min.insync.replicas?
• Why is acks=all alone insufficient for strong durability guarantees?
• Explain the difference between eager and cooperative sticky rebalancing. Why does the distinction matter operationally?
• What typically triggers an unexpected consumer group rebalance in a healthy-looking service?
• What is the difference between at-most-once, at-least-once, and exactly-once delivery semantics in Kafka?
• Why is Kafka's exactly-once semantics described as scoped to "Kafka-to-Kafka," and what does that mean for a consumer writing to a database?
• How does an idempotent producer prevent duplicate writes caused by retries?
• What's the risk of committing a consumer offset before processing a message completes?
• Design a retry and dead-letter strategy for a consumer that occasionally hits transient downstream failures.
• What happens to messages behind a "poison" message in a partition if there's no dead-letter mechanism?
• How would you decide whether to scale a Kafka consumer group by adding pods versus increasing partition count?
• What is max.poll.interval.ms, and how does exceeding it differ from missing a heartbeat (session.timeout.ms)?
• Explain why increasing a topic's partition count can affect ordering guarantees for existing keyed data.
• When would you choose non-blocking retry topics (@RetryableTopic) over a simple blocking DefaultErrorHandler with backoff?