Partitioning, bucketing, and the small-file problem

6 h
requires SPK-06

From your world. Redshift DISTKEY / SORTKEY; Athena/Iceberg partitioning and file sizing

Partition the output by a low-cardinality key

You make the same decision for Athena/Iceberg when you choose the partition column. Partition on a low-cardinality column that you filter on often. Then each partition is a unit of reasonable size that the engine can prune. If you partition on a high-cardinality column (one directory per order), you get the classic small-file explosion.

Task

Implement write_partitioned(df, out_dir). It writes df as Parquet into out_dir, physically partitioned by company_id (Hive-style company_id=<value>/ directories), with the data unchanged. Return the output path as a string.

Constraints

Partition by company_id (low cardinality: a few companies), not by a high-cardinality column such as order_id.
Overwrite any existing output.

How it is graded

Layout: out_dir contains exactly one company_id=<value>/ directory for each distinct company_id in the input, and no more. This proves that you did not partition by a high-cardinality key.
Data (DuckDB oracle): when you read the whole output back, it equals the input rows. When you read a single partition directory, it equals the rows for that company (partition pruning works).

Starter code

solution.py

"""SPK-07 / 03-partition-by-layout — starter. See README.md."""


def write_partitioned(df, out_dir):
    # TODO: write df as Parquet partitioned by company_id (low cardinality),
    # overwriting, data unchanged. Return the output path (str).
    raise NotImplementedError

Hints

Hint 1

Write with .write.partitionBy('company_id') so the output has one company_id=<value>/ directory per company — a low-cardinality, frequently-filtered column.

Hint 2

Do NOT partitionBy a high-cardinality column (order_id): that explodes into one directory per order (the small-file problem). Partition by company_id.

Hint 3

Return the output path. The test checks the company_id=.../ directory layout, that the number of partition dirs equals the number of companies, and that the data round-trips.

Run it locally

uv run de start SPK-07/03-partition-by-layout
# edit workspace/SPK-07/03-partition-by-layout/
uv run de test