Partitioning, bucketing, and the small-file problem

6 h
requires SPK-06

From your world. Redshift DISTKEY / SORTKEY; Athena/Iceberg partitioning and file sizing

Reduce files after a filter without a shuffle

This is a common pipeline shape. A DataFrame is spread across many partitions. A selective filter makes it smaller, and now you have many near-empty files. The cheap fix is coalesce. It merges existing partitions with no shuffle (a narrow transformation, no Exchange). If you use repartition here, you force a full shuffle that you do not need.

Task

Implement reduce_usd(df, num_files). It must do these two things:

Filter df to currency == 'USD'.
Reduce the result to exactly num_files partitions without adding a shuffle.

Return the resulting DataFrame.

Constraints

Use coalesce, not repartition, for the reduction (it must not add an Exchange).
num_files is smaller than the input partition count (coalesce only reduces).

How it is graded

Data (DuckDB oracle): rows equal SELECT * FROM orders WHERE currency = 'USD'.
Partitions: df.rdd.getNumPartitions() on your result equals num_files.
Plan — no added shuffle: your reduction adds no Exchange over the base plan. The test gives you a base that already has many partitions. It records the Exchange count of the base and checks that your result has the same count (coalesce is narrow). A repartition would add one and fail.

Starter code

solution.py

"""SPK-07 / 02-coalesce-no-shuffle — starter. See README.md."""


def reduce_usd(df, num_files):
    # TODO: filter to USD, then reduce to num_files partitions WITHOUT a shuffle
    # (use coalesce, not repartition).
    raise NotImplementedError

Hints

Hint 1

A selective filter leaves data spread over many partitions. To merge down to num_files WITHOUT a shuffle, use coalesce(num_files), not repartition(num_files).

Hint 2

coalesce is a NARROW transformation: it adds no Exchange to the plan. repartition would add one. The test counts Exchange nodes added by your reduction step.

Hint 3

Return the filtered-and-coalesced DataFrame with num_files partitions and the same rows as the filtered input (currency == 'USD').

Run it locally

uv run de start SPK-07/02-coalesce-no-shuffle
# edit workspace/SPK-07/02-coalesce-no-shuffle/
uv run de test