DE Mastery GitHub

Build your own mini columnar query engine · Stage 2 of 8

Projection and filter operators (Volcano model)

Add two physical operators on top of your scan, in the classic Volcano / iterator style: each operator pulls RecordBatches from its child and pushes transformed batches to its parent.

Task

Make execute() handle these plan nodes:

  • Project(child, columns) — select/reorder columns.
  • Filter(child, predicate) — keep rows where predicate is true.

Predicates are expression trees over the stage-1 API:

Filter(scan, Compare(Col("amt"), ">", Lit(25.0)))
Filter(scan, And(Compare(Col("amt"), ">=", Lit(10.0)),
                 Or(Compare(Col("cat"), "=", Lit("a")),
                    Compare(Col("cat"), "=", Lit("b")))))

Requirements

  • Evaluate predicates with pyarrow.compute: map each Compare.op to the right kernel (equal, less, greater, …) and combine And/Or with and_kleene / or_kleene so NULLs follow SQL three-valued logic.
  • Project and Filter must compose with each other and with Scan.
  • Results match DuckDB for the same WHERE / column list (G6). A Hypothesis property checks filter soundness: every surviving row satisfies the predicate and the row count matches the oracle.

Constraints

  • Stay in the iterator model — operators pull batches, they don't re-read files.
  • Earlier stage (scan) must keep passing: de test runs stages 1–2 together.

Hints

Hint 1

In the Volcano/iterator model each operator pulls batches from its child. Add Project and Filter branches to your _batches() generator so they transform the batches flowing up from the Scan.

Hint 2

Project: map the requested column names to indices and use RecordBatch.select(idxs). Filter: evaluate the predicate to a boolean mask with pyarrow.compute and call batch.filter(mask).

Hint 3

Evaluate expressions structurally: Compare(left, op, right) maps op to pc.equal/less/greater/...; And/Or combine child masks with pc.and_kleene/pc.or_kleene so NULLs follow three-valued logic like SQL.

Run it locally
uv run de start SPK-BYO/02-projection
uv run de test