DE Mastery GitHub

Apache Spark · SPK-08

Memory, caching, and Adaptive Query Execution

From your world. Redshift WLM memory/spill-to-disk and query queues; diagnosing OOM

← All exercises

Diagnose an OOM from a log excerpt

In an exam or when an alert wakes you at 2 a.m., you don't have the Spark UI. You have a log line. The real skill is to turn a log into the right fix. This exercise makes that skill gradable by a machine. You get a log excerpt string, and you return a structured diagnosis.

Task

Implement diagnose(log). It returns a dict with exactly these keys. The values come from the controlled vocabulary below:

{"location": <str>, "cause": <str>, "fix": <str>}
  • location ∈ {"driver", "executor"}
  • cause ∈ {"collect_to_driver", "skew", "spill", "overhead"}
  • fix ∈ {"avoid_collect", "salt_or_aqe_skew", "more_partitions_or_memory", "raise_overhead"}

Signatures to recognize

Log signature location cause fix
OutOfMemoryError: Java heap space with collect/toPandas/take in the stack driver collect_to_driver avoid_collect
OutOfMemoryError on an executor during a shuffle/sort where one task is far larger than the others executor skew salt_or_aqe_skew
Repeated Spilling in-memory map ... to disk (no crash) executor spill more_partitions_or_memory
Container killed ... exceeding memory limits / OOMKilled (overhead/off-heap) executor overhead raise_overhead

How it is graded

The test gives your function several representative log excerpts. For each one, it checks your returned {location, cause, fix} against the expected diagnosis. The excerpts are inline in the test. So your parser must read the log text. Do not hard-code one answer.

Starter code

solution.py
"""SPK-08 / 01-diagnose-oom — starter. See README.md."""


def diagnose(log):
    # TODO: classify the log excerpt. Return
    # {"location": ..., "cause": ..., "fix": ...} using the README vocabulary.
    raise NotImplementedError

Hints

Hint 1

Parse the log string for signatures: a collect/toPandas in the driver stack => driver OOM; one task far larger during a shuffle => executor skew; repeated 'Spilling in-memory map ... to disk' => execution-memory spill; 'Container killed ... exceeding memory limits'/'OOMKilled' => memory overhead.

Hint 2

Return a dict {'location': ..., 'cause': ..., 'fix': ...} using the exact vocabulary in the README. Base the answer on the log text, not on guesses.

Hint 3

location in {'driver','executor'}; cause in {'collect_to_driver','skew','spill','overhead'}; fix in {'avoid_collect','salt_or_aqe_skew','more_partitions_or_memory','raise_overhead'}.

Run it locally
uv run de start SPK-08/01-diagnose-oom
# edit workspace/SPK-08/01-diagnose-oom/
uv run de test