DE Mastery GitHub

Track · KFK · ~65 h

Apache Kafka

Philosophy. Kafka is a distributed, replicated, append-only log. Producers and consumers are decoupled in time: data is retained and can be replayed. It is the central nervous system between operational systems and analytics. When not to use it: request/response workflows, simple task queues with few consumers (SQS/RabbitMQ is simpler), very small data volumes, or analytics that only need a daily batch.

Kafka 4.x runs on KRaft only (no ZooKeeper). Lab profile: streaming. Clients: Python (confluent-kafka) for KFK-01..07. Java for Kafka Streams (KFK-08).

Modules

  1. 01 The log — events, state, retention and replay 3 h · 1 exercise
  2. 02 Topics, partitions, replication — leaders, ISR, acks, durability 5 h · 1 exercise
  3. 03 Producers — batching, idempotence, ordering under retries 5 h · 1 exercise
  4. 04 Consumers & groups — manual commits, at-least-once, no loss across crashes 6 h · 1 exercise
  5. 05 Transactions & exactly-once — read-process-write, read_committed, fencing 5 h · 1 exercise
  6. 06 Schema Registry & contracts 5 h · 1 exercise
  7. 07 Kafka Connect & CDC 6 h · 1 exercise
  8. 08 Kafka Streams 7 h · 1 exercise
  9. 09 Event & topic design 4 h · 1 exercise
  10. 10 Testing & observability (apps) 4 h · 1 exercise
  11. 11 Cluster configuration & partition reassignment 5 h · 1 exercise
  12. 12 Security — SASL/SCRAM authentication & ACL authorization 4 h · 1 exercise
  13. 13 Deployment architecture — multi-region DR 3 h · 1 exercise
  14. 14 Operating & troubleshooting a cluster 3 h · 1 exercise
  15. BYOBuild your own mini Kafka 10 h · 8 stages