Kafka 4.0: ZooKeeper Is Finally Gone, and Queues Arrived

Two complaints have followed Apache Kafka for its entire life, and I've made both of them myself: "why do I need to run ZooKeeper just to run a message broker?" and "Kafka is a log, not a queue — I can't use it for task-queue workloads." Kafka 4.0, released in March 2025, answers both. ZooKeeper is gone entirely, and share groups bring genuine per-record queue semantics to the broker.

I run Kafka in production and teach it for a living, and I think this is the most consequential release in the project's history — bigger than the idempotent producer, bigger than tiered storage. If you operate Kafka, it defines your next year of infrastructure work. If you're evaluating Kafka against RabbitMQ or SQS, it invalidates half the comparison tables written before 2025. Here's what actually changed, in order of operational impact.

1. ZooKeeper Is Gone — Completely

KRaft — Kafka's built-in Raft-based consensus — has been the default for new clusters for a while, but 4.0 is the line in the sand: ZooKeeper mode is removed. A 4.0 broker runs on KRaft or not at all.

Why this matters beyond one less system to run:

• Operations. No separate ZooKeeper ensemble to provision, secure, monitor, and keep version-compatible. The worst Kafka on-call nights of my career involved ZooKeeper — session storms, quorum loss cascading into broker instability. That entire failure category simply ceases to exist.
• Metadata scalability. KRaft keeps metadata in a replicated internal log and pushes it to brokers instead of round-tripping through ZooKeeper watches. Controller failover drops from tens of seconds to seconds, and the partition ceiling moves from "hundreds of thousands, carefully" to millions.
• Security surface. One less quorum system with its own ACL model, ports, and attack surface. Anyone who has audited a ZK+Kafka deployment knows how much simpler the KRaft-only picture is.

One gotcha I want to be very clear about: you cannot jump from a ZooKeeper-based cluster straight to 4.0. The migration bridge is the 3.9 release line — the last series that supports ZooKeeper — where you perform the KRaft migration first, verify stability, and only then upgrade. Budget for this as a two-stage project, not a version bump.

2. The New Consumer Rebalance Protocol Goes GA (KIP-848)

Classic consumer group rebalancing was stop-the-world: one member joins or dies, everyone revokes their partitions, the group leader recomputes the full assignment, everyone resubscribes. I've watched "rebalance storms" take a healthy consumer group to zero throughput during a routine rolling restart — lag climbing, alerts firing, everyone in the incident channel asking who deployed.

KIP-848, generally available in 4.0, replaces that with a server-driven, incremental protocol:

• The broker assigns partitions, not a client leader. The multi-round client choreography is gone.
• Rebalances are incremental. Members keep partitions that aren't moving; only the delta changes hands. A rolling restart of a 50-member group stops being an event you schedule around.
• Fewer fencing failures. The old race between zombie consumers and new assignments — the source of some truly nasty duplicate-processing bugs — is structurally tightened.

If you've ever watched consumer lag spike during a deploy for no apparent reason, this is the fix.

3. Share Groups: Kafka Learns to Be a Queue (KIP-932)

This is the headline for application developers. Classic consumer groups have a hard rule: one partition, one consumer per group. Parallelism is capped by partition count, records arrive in partition order whether you need it or not, and one poison message blocks everything behind it.

Share groups, in early access in 4.0, break that rule: multiple consumers in a share group cooperatively consume the same partitions, with the broker distributing individual records across them. On top of that you get queue-native semantics Kafka never had: per-record acknowledgement (accept, release for redelivery, reject for poison messages), delivery-attempt limits so a bad record stops being offered instead of blocking forever, and parallelism decoupled from partition count — three partitions no longer means at most three workers.

What you give up is ordering — share groups distribute records for throughput, not sequence. That's exactly the right trade for task-queue workloads: sending emails, processing webhooks, transcoding jobs, per-record enrichment. Each message is independent; nobody cares about the order.

Note that consumer groups and share groups aren't either/or — the same topic can serve both. Consumer groups for stream processing and event-driven flows where per-key ordering matters; share groups for work-queue consumption. The log is shared; the semantics are per-group.

The strategic implication is bigger than the API. The standard advice since 2015 — "RabbitMQ or SQS for task queues, Kafka for event streaming" — now needs re-litigating. One system covering both patterns means one less broker to operate, one client model, one security model. Share groups are early access, so don't bet a launch on them this quarter — but the direction is unmistakable.

4. The Cleanup Nobody Celebrates but Everyone Feels

Java 11 is the new floor for brokers and clients — if you still have Java 8 consumers, this is your forcing function. Ancient wire-protocol versions and pre-0.10 message formats are removed, so very old clients must be upgraded before the brokers. And a pile of long-deprecated configs and tools is gone — clusters upgraded continuously since the 1.x era tend to carry config fossils that 4.0 will refuse to boot with. Read the removal notes before you plan the window.

How I'd Get to 4.0

My path for a production cluster, in order:

1. Inventory clients first. Broker support is meaningless if a forgotten Flink job from 2021 speaks a removed protocol version. Upgrade producers and consumers to a supported line before touching brokers.
2. Migrate to KRaft on 3.9. The riskiest step. Own maintenance window, rollback plan, soak time. Do not bundle it with the 4.0 upgrade.
3. Upgrade to 4.0. With KRaft stable underneath, this part is comparatively routine.
4. Opt consumer groups into the new rebalance protocol deliberately, per group, watching rebalance metrics during rollout.
5. Pilot share groups on something non-critical — an internal notification or webhook pipeline is ideal — to build operational intuition before the feature goes GA in a future release.

If you're newer to Kafka and want the fundamentals before the upgrade mechanics, start with Kafka fundamentals, then reliability and scaling, and the Kafka roadmap. For the deep-dive, production-with-Spring-Boot version, that's literally what the Kafka Ecosystem Masterclass is for.

The Bottom Line

Kafka 4.0 is the project finishing its own decade-long to-do list: KRaft-only operation removes the worst operational burden, the KIP-848 rebalance protocol removes the worst runtime behavior, and share groups remove the most common reason teams reached for a second broker. Plan the two-stage upgrade, audit your clients, and start thinking about which of your RabbitMQ or SQS workloads you'd consolidate back onto the log.

References

• Apache Kafka 4.0 — downloads and release notes
• KIP-500 — Replace ZooKeeper with a Self-Managed Metadata Quorum
• KIP-848 — The Next Generation of the Consumer Rebalance Protocol
• KIP-932 — Queues for Kafka (share groups)
• ZooKeeper to KRaft migration guide (Confluent)