Spark architecture

From your world. Redshift leader/compute nodes vs. Spark driver/executors

Count the tasks a scan stage runs, and place them on executors

When you open the Spark UI, each stage shows a list of tasks. It also shows which executor ran each task. This exercise makes that concrete. The first (scan) stage of a job runs one task per partition. The scheduler spreads those tasks across the executors.

Implement two functions in solution.py:

`task_count(df) -> int`

Return the number of tasks that the first stage of df would run. This equals the number of partitions of the underlying RDD of the DataFrame. Read it from the DataFrame at runtime. Do not guess.

`tasks_per_executor(num_tasks, num_executors) -> list[int]`

Return a list of length num_executors. It gives how many of the num_tasks tasks each executor runs when the scheduler balances them as evenly as possible. The counts must:

sum to num_tasks,
differ by at most 1 between any two executors,
be in non-increasing order (busiest executor first).

Example: tasks_per_executor(7, 3) == [3, 2, 2]. Example: tasks_per_executor(6, 3) == [2, 2, 2].

Constraints:

num_executors >= 1, num_tasks >= 0.
task_count must reflect the actual partitioning of the DataFrame (including after repartition).

Starter code

solution.py

"""SPK-01 / 02-tasks-and-executors — starter. See README.md."""


def task_count(df):
    # TODO: number of tasks in the first stage = number of RDD partitions of df.
    raise NotImplementedError


def tasks_per_executor(num_tasks, num_executors):
    # TODO: balance num_tasks across num_executors; busiest first.
    raise NotImplementedError

Hints

Hint 1

A DataFrame's underlying RDD has one partition per task in its first stage: df.rdd.getNumPartitions().

Hint 2

repartition(n) sets the partition count to n; that becomes the task count of the next stage. coalesce only reduces it.

Hint 3

To place t tasks on e executors as evenly as possible, each executor gets either floor(t/e) or ceil(t/e); exactly (t mod e) executors get the ceil.

Run it locally

uv run de start SPK-01/02-tasks-and-executors
# edit workspace/SPK-01/02-tasks-and-executors/
uv run de test