Memory, caching, and Adaptive Query Execution

6 h
requires SPK-07

From your world. Redshift WLM memory/spill-to-disk and query queues; diagnosing OOM

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz SPK-08.

1A job dies with java.lang.OutOfMemoryError: Java heap space. The stack trace points at a collect()/toPandas() call. Where did it OOM and what is the fix? AAn executor OOMed. Add more executors. BThe driver OOMed when it pulled the whole dataset to itself. Do not collect the full data. Reduce/aggregate first or write to storage. CThe shuffle spilled. Raise shuffle.partitions. DStorage memory was full. Call unpersist.

3What does df.cache() do, and when is it worth it? AIt writes the DataFrame to disk immediately. It is always worth it. BIt marks the DataFrame to be persisted (MEMORY_AND_DISK) on the next action. It is worth it only when you reuse the DataFrame across multiple actions. CIt caches only the schema. DIt forces a broadcast join.

4Which are genuine Adaptive Query Execution (AQE) optimizations that you can see in the final plan? (select all) (select all that apply) ACoalescing many small post-shuffle partitions into fewer partitions of the right size. BRuntime skew-join handling that splits an oversized shuffle partition. CDynamically switching a sort-merge join to a broadcast join when a side turns out small. DPushing filters into the Parquet scan.

5Why must you look at the FINAL executed plan (after an action) to see AQE's effects? Why is the plan from explain() before the run not enough? Aexplain() is broken with AQE. BAQE re-optimizes at runtime using actual shuffle statistics. Its changes (coalesced partitions, skew handling, join switch) appear only in the final plan, after the first stage runs. CAQE only affects reads. DThe initial plan already shows AQE results.

6What do repeated log lines Spilling in-memory map ... to disk during an aggregation indicate? AA driver OOM. BExecution-memory pressure. The aggregation does not fit in memory and spills to disk (correct but slow). Fix it with more partitions/memory or a narrower aggregation. CCaching succeeded. DThe job failed.

7You cache a DataFrame, but you see no speedup and higher memory use. What likely happened? ACaching always speeds things up. The measurement is wrong. BThe DataFrame is used only once (no reuse). Caching only uses storage memory with no benefit, and it may force eviction/spill elsewhere. Ccache() requires DISK_ONLY to work. DYou must call collect() twice for cache to apply.

8An executor OOMs during a shuffle. One task processed 50x more data than the others. Which fix targets the actual cause? ACollect less data to the driver. BThis is skew. Salt the hot key (SPK-06), add partitions, or enable AQE skew-join handling so that Spark splits the oversized partition. CCache the DataFrame. DSwitch to DISK_ONLY storage.