Kafka Fundamentals: Brokers, Topics, Partitions, and Offsets

Kafka Fundamentals

Apache Kafka is the backbone of event-driven backend systems — it decouples producers from consumers, absorbs traffic spikes, and gives you a durable, replayable log of everything that happened in your system. This guide covers the mental model every backend engineer needs before writing a single line of producer or consumer code: how brokers, topics, partitions, and offsets fit together, and why the partitioning scheme you choose determines both your throughput ceiling and your ordering guarantees.

1. Why Event Streaming, Not Just a Queue

Traditional message queues (RabbitMQ, ActiveMQ, SQS) are built around a consume-and-delete model: a message is removed from the queue once it's acknowledged. Kafka inverts this. It is a distributed, append-only commit log — messages are retained for a configured period (or forever, for compacted topics) regardless of whether they've been read. Multiple independent consumers can replay the same data at different times without interfering with each other.

Message queue vs Kafka

2. Brokers and Clusters

A broker is a single Kafka server process — it stores data on disk, serves produce/fetch requests, and participates in replication. A cluster is a group of brokers coordinating together. Modern Kafka (3.x+ with KRaft) manages cluster metadata via a built-in Raft-based quorum controller, replacing the older ZooKeeper-based coordination.

Each broker owns a subset of partitions as leader and holds follower replicas of others. Clients always produce to and fetch from the current partition leader — never directly from a follower (except with rack-aware "follower fetching" for read locality, a separate optimization).

3. Topics and Partitions

A topic is a named, logical stream of events — orders, payment-events, user-signups. Under the hood, a topic is split into one or more partitions, and each partition is an independently ordered, append-only log stored as a set of segment files on disk.

Why partitioning matters

Partitions are the unit of parallelism in Kafka:

• Producer throughput scales because different partitions can be written to different brokers concurrently.
• Consumer throughput scales because each partition can only be read by one consumer within a given consumer group at a time — more partitions means more consumers can work in parallel.
• Ordering is guaranteed only within a partition, never across partitions of the same topic.

# Create a topic with 6 partitions and replication factor 3
kafka-topics.sh --create \
  --topic orders \
  --bootstrap-server broker1:9092 \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config min.insync.replicas=2

4. Offsets and the Log

Every message within a partition gets a monotonically increasing offset — its position in that partition's log. Offsets are partition-local: partition 0's offset 42 and partition 1's offset 42 are unrelated messages.

Anatomy of a log segment

On disk, each partition's log is split into segments — files rolled over by size or time (log.segment.bytes, log.segment.ms). This makes retention cheap: expiring old data means deleting whole segment files, not scanning and rewriting a log.

# Inspect committed offsets for a consumer group
kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group billing-service
 
# GROUP           TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# billing-service  orders  0          1042            1050            8
# billing-service  orders  1          998             998             0

LAG is the single most important operational metric in Kafka consumption — it tells you how far behind a consumer group is from the latest produced data.

5. Producers

A producer sends records to a topic. Kafka does not require you to pick a partition explicitly — a partitioner decides which partition a record lands in, based on whether the record has a key.

@Configuration
public class KafkaProducerConfig {
 
    @Bean
    public ProducerFactory<String, OrderEvent> producerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.LINGER_MS_CONFIG, 5);
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 32_768);
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
        return new DefaultKafkaProducerFactory<>(props);
    }
 
    @Bean
    public KafkaTemplate<String, OrderEvent> kafkaTemplate(
            ProducerFactory<String, OrderEvent> producerFactory) {
        return new KafkaTemplate<>(producerFactory);
    }
}

How keys determine partitioning

• Keyed records (key = orderId) always hash to the same partition — this is how you get per-key ordering: every event for a given order lands in the same partition, in send order.
• Unkeyed records use Kafka's sticky partitioner (since 2.4+), which batches records for one partition at a time before switching — improving batching efficiency versus strict round-robin, at the cost of no ordering guarantee across records.

@Service
public class OrderEventPublisher {
 
    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;
 
    public OrderEventPublisher(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }
 
    public void publish(OrderEvent event) {
        // Keying by orderId guarantees all events for this order
        // are processed in order by any single consumer.
        kafkaTemplate.send("orders", event.orderId(), event)
            .whenComplete((result, ex) -> {
                if (ex != null) {
                    log.error("Failed to publish order event {}", event.orderId(), ex);
                } else {
                    log.debug("Published to partition {} offset {}",
                        result.getRecordMetadata().partition(),
                        result.getRecordMetadata().offset());
                }
            });
    }
}

Batching and linger

Producers don't send one record per network call — they buffer records per partition into batches, controlled by batch.size (max bytes per batch) and linger.ms (max time to wait for a batch to fill before sending anyway). This trade-off is the classic latency-vs-throughput knob.

6. Consumers and Consumer Groups

A consumer group is a set of consumer instances that cooperatively consume a topic, splitting the partitions among themselves. Each partition is assigned to exactly one consumer within a group at any moment — this is what makes horizontal scaling of consumption safe without duplicate processing within the group.

The golden rule of consumer scaling

Consumers in a group ≤ partitions in the topic. Extra consumers beyond the partition count sit completely idle — they never get assigned a partition.

Independent consumer groups get independent copies

Each consumer group tracks its own committed offsets. Two different groups reading the same topic don't affect each other at all — this is how Kafka supports fan-out to multiple independent applications from one topic.

@Component
public class OrderEventListener {
 
    @KafkaListener(
        topics = "orders",
        groupId = "billing-service",
        concurrency = "3"  // spins up 3 consumer threads within this app instance
    )
    public void onOrderEvent(
            @Payload OrderEvent event,
            @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
            @Header(KafkaHeaders.OFFSET) long offset) {
        log.info("Processing order {} from partition {} offset {}",
            event.orderId(), partition, offset);
        billingService.charge(event);
    }
}

7. Ordering Guarantees

Kafka's ordering guarantee is precise and easy to misstate: records are ordered within a partition, in the order the producer sent them (assuming no retries reorder them — see idempotent producers in the next guide), and consumed in that same order by whichever consumer owns the partition. There is no ordering guarantee across partitions, even within the same topic.

This is why key selection is an ordering decision, not just a load-balancing one. If your business logic requires "all events for order X must be processed in the order they occurred," you must key by orderId (or whatever entity needs ordering) — never rely on global ordering across a multi-partition topic, because it does not exist.

8. Production Observations

• Partition count is a one-way door for scaling up. Choose it based on your projected peak consumer parallelism, not today's traffic — you can't shrink it later without recreating the topic.
• Lag is your primary health signal. Alert on consumer group lag (in messages and in time-behind), not just on broker uptime — a healthy cluster with a stuck consumer looks fine from the broker's perspective.
• Hot partitions kill throughput silently. A poorly chosen key (low cardinality, skewed distribution) caps your effective parallelism even with plenty of partitions provisioned.
• bootstrap.servers needs only a few broker addresses — full cluster topology is discovered dynamically, so you don't need every broker listed, just enough for initial connectivity redundancy.
• Retention and consumer downtime interact. If a consumer group can be down longer than retention.ms, it will lose data on resume (via auto.offset.reset) unless you explicitly monitor and alert on approaching retention boundaries.

Key takeaways

• Kafka is a durable, replayable, partitioned log — not a delete-on-read queue; multiple consumer groups can independently replay the same topic.
• Partitions are the unit of both producer throughput and consumer parallelism; consumer count beyond partition count is wasted capacity.
• Ordering is guaranteed only within a partition; keyed records deterministically hash to the same partition, giving per-key ordering.
• Offsets are partition-local and tracked per consumer group in __consumer_offsets; lag (LOG-END-OFFSET minus CURRENT-OFFSET) is the key operational metric.
• Producer batching (linger.ms, batch.size, compression.type) trades latency for throughput and compression efficiency.
• Increasing partitions on an existing topic is safe for scaling but silently changes the key-to-partition mapping — plan around it.
• Retention windows and consumer downtime must be reasoned about together, or you risk silent data loss via auto.offset.reset.

Interview Questions

• What is the difference between a Kafka topic and a partition? Why do partitions exist?
• How does Kafka guarantee ordering, and why is that guarantee scoped to a single partition?
• What determines which partition a producer record is sent to? How does keying affect this?
• What happens if you add more consumer instances to a group than there are partitions?
• Explain the difference between a topic's log-end offset and a consumer group's committed offset.
• What is consumer lag, and how would you monitor and alert on it in production?
• Why can you not decrease the partition count of an existing topic? What happens when you increase it?
• What is a hot partition, and how would you diagnose and fix one?
• How does Kafka differ fundamentally from a traditional message queue like RabbitMQ or SQS?
• What role does linger.ms and batch.size play in producer performance? What trade-off do they represent?
• What happens to a consumer group's offsets if it's down longer than the topic's retention period?
• Explain how the sticky partitioner behaves differently from strict round-robin for unkeyed records.
• What is log compaction, and when would you choose cleanup.policy=compact over time/size-based retention?
• Why does KRaft (Kafka's built-in Raft consensus) replace ZooKeeper in modern Kafka clusters, at a high level?