DE Mastery GitHub

Apache Spark · SPK-01

Spark architecture

  • 5 h

From your world. Redshift leader/compute nodes vs. Spark driver/executors

← All exercises

Configure a local Spark session and read its runtime config

The spark session that the tests give you runs in local mode. Your job is to read its runtime configuration and calculate how parallel a job would actually be. This is the most useful piece of architecture intuition.

Implement three functions in solution.py:

local_cores(spark) -> int

Return the number of executor threads (task slots) that this local session started with. This is the N in local[N]. Read it from the master string of the session at runtime. Do not hard-code it. Do not use defaultParallelism. Setting spark.default.parallelism can override that value, so it does not always equal N.

shuffle_partitions(spark) -> int

Return the configured number of shuffle partitions (spark.sql.shuffle.partitions) as an int.

peak_parallelism(num_partitions, cores) -> int

A stage has num_partitions partitions. It runs on a cluster with cores total cores. Return the maximum number of tasks that run at the same time. Remember: one task processes one partition. You cannot run more tasks at once than you have cores.

Constraints:

  • Do not hard-code 2 or 4. Read the real values from the session.
  • peak_parallelism must be a pure function of its two integer arguments.
  • Inputs to peak_parallelism are non-negative integers.

Starter code

solution.py
"""SPK-01 / 01-configure-session — starter.

Implement the three functions so the tests pass. See README.md.
"""


def local_cores(spark):
    # TODO: return the N in local[N], read from the session's master string.
    raise NotImplementedError


def shuffle_partitions(spark):
    # TODO: return spark.sql.shuffle.partitions as an int.
    raise NotImplementedError


def peak_parallelism(num_partitions, cores):
    # TODO: max tasks running at once = min(num_partitions, cores).
    raise NotImplementedError

Hints

Hint 1

spark.sparkContext.master is a string like 'local[2]'. The number inside the brackets is the executor-thread (core) count.

Hint 2

Parse the N out of local[N]; handle local[*] by falling back to os.cpu_count(). Do NOT use defaultParallelism — it can be overridden and won't equal N.

Hint 3

shuffle_partitions: spark.conf.get(...) returns a str, so wrap it in int(). peak_parallelism is just min(num_partitions, cores).

Run it locally
uv run de start SPK-01/01-configure-session
# edit workspace/SPK-01/01-configure-session/
uv run de test