Roadmap
Analytics engineer to data platform engineer
Follow the line from the top. Click a node to open it, and tick the circle when you finish it. Your progress is kept in this browser only.
Track Module Build your own Done
0 of 71 done
- 01 Spark architecture 5 h
- 02 Lazy evaluation and the DAG 5 h
- 03 The DataFrame API 7 h
- 04 Spark SQL and data sources 5 h
- 05 Catalyst and the physical plan 6 h
- 06 Joins and aggregations at scale 6 h
- 07 Partitioning, bucketing, and the small-file problem 6 h
- 08 Memory, caching, and Adaptive Query Execution 6 h
- 09 UDFs and the Pandas API 4 h
- 10 Structured Streaming I — the micro-batch engine and exactly-once 6 h
- 11 Structured Streaming II — watermarks, state, and streaming SCD2 7 h
- 12 Testing and production Spark 7 h
- BYO Build your own mini columnar query engine 12 h
- 01 The Iceberg metadata tree 4 h
- 02 Commits & concurrency 4 h
- 03 Catalogs, branching & write-audit-publish 4 h
- 04 Row-level changes — SCD2, CoW & MoR 5 h
- 05 Table layout & maintenance 5 h
- 06 Delta Lake 4 h
- 07 Hudi & the format landscape 3 h
- 08 Engines on the lake 6 h
- BYO Build your own mini table format 8 h
- 01 The log — events, state, retention and replay 3 h
- 02 Topics, partitions, replication — leaders, ISR, acks, durability 5 h
- 03 Producers — batching, idempotence, ordering under retries 5 h
- 04 Consumers & groups — manual commits, at-least-once, no loss across crashes 6 h
- 05 Transactions & exactly-once — read-process-write, read_committed, fencing 5 h
- 06 Schema Registry & contracts 5 h
- 07 Kafka Connect & CDC 6 h
- 08 Kafka Streams 7 h
- 09 Event & topic design 4 h
- 10 Testing & observability (apps) 4 h
- 11 Cluster configuration & partition reassignment 5 h
- 12 Security — SASL/SCRAM authentication & ACL authorization 4 h
- 13 Deployment architecture — multi-region DR 3 h
- 14 Operating & troubleshooting a cluster 3 h
- BYO Build your own mini Kafka 10 h
- 01 Philosophy & architecture 4 h
- 02 DataStream API 6 h
- 03 Time & windows 6 h
- 04 Flink SQL & dynamic tables 6 h
- 05 Joins — regular, interval, temporal (point-in-time) & lookup 5 h
- 06 State, checkpoints & exactly-once — surviving a TaskManager crash 7 h
- 07 Process functions — timers, keyed state & broadcast rules 5 h
- 08 Table API & PyFlink — one pipeline, two languages 4 h
- 09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER 5 h
- 10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution 6 h
- 11 Performance & state at scale — backpressure, parallelism, redistribution, skew 6 h
- 12 Deploy & operate — upgrade a stateful job via savepoint without losing state 5 h
- BYO Build your own mini stream processor 10 h
- 01 Architecture history — Lambda → Kappa → lakehouse → streamhouse 3 h
- 02 Apache Paimon — primary-key tables, merge engines, partial-update 6 h
- 03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency 5 h
- 04 Iceberg as streaming storage — commit interval vs small files 4 h
- 05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join 6 h
- 06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness 4 h
- 07 Serving layer — OLAP on the streamhouse 4 h
- 08 Latency tiers & cost — seconds, minutes, hours 3 h
You can now design and run batch, streaming and streamhouse pipelines.