Partitioning, bucketing, and the small-file problem

6 h
requires SPK-06

From your world. Redshift DISTKEY / SORTKEY; Athena/Iceberg partitioning and file sizing

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz SPK-07.

1What is the key difference between repartition(n) and coalesce(n)? AThey are identical. Brepartition(n) does a full shuffle to produce n balanced partitions (and can increase the count). coalesce(n) merges existing partitions with no shuffle and can only reduce the count. Ccoalesce always shuffles. repartition never does. Dcoalesce can only increase the partition count.

5How does bucketing let two tables join without a shuffle at read time? AIt caches both tables. BbucketBy pre-shuffles each table on the key into a fixed number of buckets at write time. When you later join the tables on that key, matching keys are already colocated. CIt broadcasts both tables. DIt sorts the rows alphabetically.

6Why is coalesce(1) on a very large DataFrame dangerous? AIt raises an error. BIt forces all data through a single task/partition. This removes parallelism and risks OOM/slowness. You should calculate the file count from the data size instead. CIt shuffles twice. DIt produces corrupted files.

7repartition(n) vs partitionBy(col) — what is the correct distinction? AThey are the same operation. Brepartition sets the in-memory task/partition layout during computation. partitionBy sets the on-disk directory layout of the written output. They meet at write time. CpartitionBy shuffles in memory. repartition writes directories. DBoth only affect reads.