DE Mastery GitHub

Apache Spark · SPK-06

Joins and aggregations at scale

From your world. Redshift DISTSTYLE ALL ≈ broadcast; distribution keys and skew

← All exercises

Broadcast the small dimension

Add the name of the parent company to every order. orders is the large fact. The dimension contains only the parent companies. These are the companies that no one acquired (acquired_by IS NULL). The orders of the acquired company have no row in this dimension. They must survive the join with a NULL name. The right strategy is a broadcast hash join. Spark sends the tiny dimension to every executor, and the fact streams through with no shuffle. This is exactly like Redshift DISTSTYLE ALL on a small dimension.

The test deliberately disables auto-broadcast (autoBroadcastJoinThreshold = -1). So you must request the broadcast explicitly. Then you can verify the choice in the plan.

Task

Implement enrich_orders(orders, companies). It must do these steps:

  1. Restrict companies to parent companies (acquired_by IS NULL).
  2. Left-join orders to that dimension on company_id (keep every order).
  3. Return order_id, company_id, name (the parent company name, NULL for orders of the acquired company).

Constraints

  • Explicitly broadcast the small side: F.broadcast(...).
  • Use a left join, so that you do not drop the orders of the acquired company.

How it is graded

  • Data (DuckDB oracle): equals orders LEFT JOIN (companies WHERE acquired_by IS NULL) on company_id, projected to the three columns (orders of the acquired company have name = NULL).
  • Plan: BroadcastHashJoin is present and SortMergeJoin is absent.
  • The test does not check row order or column order.

Starter code

solution.py
"""SPK-06 / 01-broadcast-join — starter. See README.md."""


def enrich_orders(orders, companies):
    # TODO: restrict companies to parents (acquired_by IS NULL), then LEFT-join
    # orders to it on company_id, broadcasting the SMALL side explicitly
    # (auto-broadcast is disabled in the test). Return order_id, company_id, name.
    raise NotImplementedError

Hints

Hint 1

Join orders to companies on company_id with a LEFT join (keep every order). The test disables auto-broadcast, so wrap the small side explicitly: F.broadcast(companies).

Hint 2

Broadcast the SMALL side (companies), never the large fact. The plan must show BroadcastHashJoin and no SortMergeJoin.

Hint 3

Return order_id, company_id, name (the company name). Row/column order are not checked; the data is compared to a DuckDB LEFT JOIN oracle.

Run it locally
uv run de start SPK-06/01-broadcast-join
# edit workspace/SPK-06/01-broadcast-join/
uv run de test