Apache Spark · Build your own · SPK-BYO
Build your own mini columnar query engine
- 12 h
- 8 stages
Build the real thing from scratch, one stage at a time. Each stage has its own tests, and you move on when it passes.
- 01Scan Parquet in row-group batches
- 02Projection and filter operators (Volcano model)
- 03Predicate pushdown using Parquet row-group statistics
- 04Hash aggregate with partial vs. final aggregation
- 05Hash join and broadcast join
- 06Partitioned execution with a hash shuffle (multiprocessing)
- 07Rule-based optimizer (filter pushdown, projection pruning)
- 08Spill to disk under a memory budget (optional)
Run it locally
uv run de start SPK-BYO/01
uv run de test