Build your own mini columnar query engine · Stage 2 of 8
Projection and filter operators (Volcano model)
Add two physical operators on top of your scan, in the classic Volcano /
iterator style: each operator pulls RecordBatches from its child and pushes
transformed batches to its parent.
Task
Make execute() handle these plan nodes:
Project(child, columns)— select/reorder columns.Filter(child, predicate)— keep rows wherepredicateis true.
Predicates are expression trees over the stage-1 API:
Filter(scan, Compare(Col("amt"), ">", Lit(25.0)))
Filter(scan, And(Compare(Col("amt"), ">=", Lit(10.0)),
Or(Compare(Col("cat"), "=", Lit("a")),
Compare(Col("cat"), "=", Lit("b")))))
Requirements
- Evaluate predicates with
pyarrow.compute: map eachCompare.opto the right kernel (equal,less,greater, …) and combineAnd/Orwithand_kleene/or_kleeneso NULLs follow SQL three-valued logic. ProjectandFiltermust compose with each other and withScan.- Results match DuckDB for the same
WHERE/ column list (G6). A Hypothesis property checks filter soundness: every surviving row satisfies the predicate and the row count matches the oracle.
Constraints
- Stay in the iterator model — operators pull batches, they don't re-read files.
- Earlier stage (scan) must keep passing:
de testruns stages 1–2 together.
Hints
Hint 1
In the Volcano/iterator model each operator pulls batches from its child. Add Project and Filter branches to your _batches() generator so they transform the batches flowing up from the Scan.
Hint 2
Project: map the requested column names to indices and use RecordBatch.select(idxs). Filter: evaluate the predicate to a boolean mask with pyarrow.compute and call batch.filter(mask).
Hint 3
Evaluate expressions structurally: Compare(left, op, right) maps op to pc.equal/less/greater/...; And/Or combine child masks with pc.and_kleene/pc.or_kleene so NULLs follow three-valued logic like SQL.
uv run de start SPK-BYO/02-projection
uv run de test