DE Mastery GitHub

Apache Spark · SPK-12

Testing and production Spark

From your world. dbt tests and CI

← All exercises

Config precedence and a testable transform

Production Spark code is configured, not hard-coded. Build the configuration layer and the pure transform. Together, they make a job portable and testable.

Implement two things in solution.py:

1. build_config(env, overrides=None) -> dict[str, str]

Merge three layers. Each layer has a higher precedence than the one before:

  1. BASE defaults (always present): at least
    • spark.sql.shuffle.partitions = "200"
    • spark.sql.adaptive.enabled = "false"
    • spark.sql.session.timeZone = "UTC"
  2. per-env profile for env in {"dev", "prod"}:
    • dev: keep BASE, but spark.sql.shuffle.partitions = "8".
    • prod: spark.sql.adaptive.enabled = "true" and spark.sql.shuffle.partitions = "400".
  3. overrides: an optional dict that wins over everything.

Rules:

  • The result always contains every BASE key. Profiles and overrides change only the keys that they set.
  • overrides has the highest precedence. The env profile beats BASE.
  • Do not put secrets or absolute paths in BASE or the profiles. Anything that is specific to an environment must arrive through overrides. The test checks that BASE/dev/prod contain no key that looks like a secret.

2. revenue_report(orders_df) -> DataFrame

A pure transform. You get orders_df(company_id, order_item_id, line_amount). Return one row per company_id with revenue = sum(line_amount). Do no I/O and use no hard-coded paths. It is only a DataFrame -> DataFrame function, so you can unit-test it.

The test checks the config precedence rules (dev vs prod vs overrides, BASE always present, no secrets). It also compares revenue_report against a DuckDB oracle.

See emr_serverless/README.md in this exercise for the optional, ungraded 💲 50 GB EMR Serverless run (cost estimate, required project=de-mastery tag, and a teardown script). It does not call AWS for you.

Starter code

solution.py
"""SPK-12 / 02-project-config — starter. See README.md."""

from pyspark.sql import functions as F


def build_config(env, overrides=None):
    # TODO: BASE defaults < per-env profile (dev/prod) < overrides. No secrets in BASE/profiles.
    raise NotImplementedError


def revenue_report(orders_df):
    # TODO: pure transform: per company_id, revenue = sum(line_amount).
    raise NotImplementedError

Hints

Hint 1

build_config(env, overrides) merges three layers with increasing precedence: BASE defaults < per-env profile (dev/prod) < explicit overrides. Never put secrets in the defaults; read them from the provided overrides/env only.

Hint 2

prod must set spark.sql.adaptive.enabled = 'true' and a larger spark.sql.shuffle.partitions than dev; both inherit the BASE keys they don't override. Return a flat dict of string values.

Hint 3

revenue_report(orders_df) is a pure DataFrame->DataFrame transform: per company_id, revenue = sum(line_amount). Keep it free of I/O and hard-coded paths so it is unit-testable; the test compares it to a DuckDB oracle.

Run it locally
uv run de start SPK-12/02-project-config
# edit workspace/SPK-12/02-project-config/
uv run de test