DE Mastery GitHub

Track · SPK · ~70 h

Apache Spark

Philosophy. Spark is a general distributed compute engine. It builds a lazy, optimizable plan (DAG) and runs it as stages split at shuffle boundaries. It is strong for large batch jobs, complex transformations and micro-batch streaming. When not to use it: data under ~100 GB (DuckDB/Polars is faster and simpler), millisecond-latency streaming (use Flink), simple SQL-only transformations that a warehouse already handles well.

Language: PySpark (as on the exam). Scala is read-only: you learn to read stack traces and Scala docs, not to write Scala. Lab profile: core.

Modules

  1. 01 Spark architecture 5 h · 2 exercises
  2. 02 Lazy evaluation and the DAG 5 h · 2 exercises
  3. 03 The DataFrame API 7 h · 3 exercises
  4. 04 Spark SQL and data sources 5 h · 3 exercises
  5. 05 Catalyst and the physical plan 6 h · 3 exercises
  6. 06 Joins and aggregations at scale 6 h · 3 exercises
  7. 07 Partitioning, bucketing, and the small-file problem 6 h · 3 exercises
  8. 08 Memory, caching, and Adaptive Query Execution 6 h · 3 exercises
  9. 09 UDFs and the Pandas API 4 h · 3 exercises
  10. 10 Structured Streaming I — the micro-batch engine and exactly-once 6 h · 2 exercises
  11. 11 Structured Streaming II — watermarks, state, and streaming SCD2 7 h · 3 exercises
  12. 12 Testing and production Spark 7 h · 2 exercises
  13. BYOBuild your own mini columnar query engine 12 h · 8 stages