DE Mastery GitHub

Track · FLK · ~65 h

Apache Flink

Philosophy. Flink is a stateful stream processor. Streaming is the primitive, and batch is a bounded stream. It gives low latency, true event-time semantics, large managed state and exactly-once results through distributed snapshots (checkpoints). When not to use it: daily batch work (Spark/dbt is simpler), stateless message routing (Kafka Connect/SMTs are enough), or teams without the skills to operate stateful jobs.

Flink 2.x. Lab profile: flink (+ streaming). APIs: Flink SQL first (closest to your SQL skills), then Table API and DataStream API in Java, and PyFlink where it fits.

Modules

  1. 01 Philosophy & architecture 4 h · 1 exercise
  2. 02 DataStream API 6 h · 1 exercise
  3. 03 Time & windows 6 h · 2 exercises
  4. 04 Flink SQL & dynamic tables 6 h · 1 exercise
  5. 05 Joins — regular, interval, temporal (point-in-time) & lookup 5 h · 1 exercise
  6. 06 State, checkpoints & exactly-once — surviving a TaskManager crash 7 h · 1 exercise
  7. 07 Process functions — timers, keyed state & broadcast rules 5 h · 1 exercise
  8. 08 Table API & PyFlink — one pipeline, two languages 4 h · 1 exercise
  9. 09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER 5 h · 1 exercise
  10. 10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution 6 h · 1 exercise
  11. 11 Performance & state at scale — backpressure, parallelism, redistribution, skew 6 h · 1 exercise
  12. 12 Deploy & operate — upgrade a stateful job via savepoint without losing state 5 h · 1 exercise
  13. BYOBuild your own mini stream processor 10 h · 8 stages