Apache Kafka Roadmap
Mastering Distributed Event Streaming from Zero to Production
Your Journey at a Glance
💡 How to use this roadmap
Work through each phase in order. Click on a skill to expand it — you'll find a description and curated resources. Don't rush; understanding beats speed. Complete one phase before moving to the next.
Foundations & The Distributed Log
Understand why Kafka exists. Learn the fundamental difference between a Message Queue and a Distributed Log.
Producer Mastery
Sending data reliably and efficiently. Learn how to tune producers for latency vs throughput.
Consumer Deep Dive
Reading data at scale. Learn how to manage state, handle failures, and scale consumption through groups.
Storage Internals & Performance
How Kafka stores data on disk. Learn about segments, compaction, and zero-copy optimization.
Reliability & Exactly-Once (EOS)
The peak of Kafka expertise. Learn how to achieve transactional consistency across producers and consumers.
The Ecosystem: Connect & Schema Registry
Moving data to and from other systems without writing custom code. Managing data quality at scale.
Stream Processing: Streams & ksqlDB
Processing data as it arrives. Learn how to join streams, manage state, and run SQL on top of Kafka.
Operations & Real-World Patterns
Running Kafka in production and designing robust event-driven systems.
Roadmap Complete!
You now have the foundations of a production-ready Java engineer. Apply by building real projects.
Build a Real-Time Fraud Detection & Notification System
Architect a complete event-driven system that processes financial transactions, detects fraud using stateful streaming, and notifies users with millisecond latency.
What you'll build
- Transactional Outbox pattern to ingest events from a PostgreSQL database reliably
- Schema Registry to enforce Avro contracts across Producer and Consumer services
- Kafka Streams app to join high-volume Transaction stream with account metadata (KTable)
- ksqlDB for real-time aggregation and anomaly detection (e.g., more than 5 payments in 1 minute)
- Dead Letter Queue (DLQ) for malformed or unprocessable payment events
- MirrorMaker 2 to replicate critical fraud events to a secondary disaster recovery cluster
Tech stack
Key highlights
- ✦Demonstrates mastery of Exactly-Once Semantics (EOS) and stateful streaming
- ✦Covers the full ecosystem: Connect, Registry, and Streaming CLI
- ✦Production-ready: includes DLQ patterns, monitoring, and disaster recovery design
Real-World Scenarios
Practical case studies where these skills are applied.
01The Thundering Herd: Consumer Group Rebalance
The Problem
A cluster of 50 consumers experienced a massive heartbeat timeout, triggering a constant loop of rebalancing that brought message processing to a halt.
The Solution
Optimized `max.poll.records` and `max.poll.interval.ms` to match actual processing time. Switched to 'Cooperative Sticky' assignor to allow incremental rebalancing without stopping the world.
Outcome
02Data Loss in a High-Throughput Cluster
The Problem
During a broker failure, some produce requests were lost despite the producer receiving a success response, leading to inconsistent financial ledgers.
The Solution
Changed `acks` from `1` to `all` and set `min.insync.replicas` to 2. Enabled `idempotence` on the producer to prevent duplicate messages during retries.
Outcome
03The Poison Pill: Deserialization Failure
The Problem
A single malformed message in a topic caused all consumers in a group to crash repeatedly, leading to a permanent processing block (Poison Pill).
The Solution
Implemented a 'Dead Letter Queue' (DLQ) pattern in the consumer logic. Added a global error handler that catches deserialization errors and routes the raw bytes to an audit topic.
Outcome