Testing and production Spark

7 h
💲
requires SPK-08

From your world. dbt tests and CI

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz SPK-12.

1A unit test passes your correct transformation. What else must be true for the test to be trustworthy? ANothing. Passing the correct code is sufficient. BIt must also FAIL plausible wrong versions (mutants) of the transformation. CIt must assert the exact column order. DIt must run on full-scale production data.

3Which assertions are appropriate in a Spark transformation test? (select all) (select all that apply) ARow-level equality to an independently-computed expected result (oracle), ignoring row order. BAn invariant such as sum(debits) == sum(credits). CThe exact physical column order of the output. DA plan property like 'no BatchEvalPython' for a performance requirement.

4Why structure a pipeline as pure DataFrame -> DataFrame transforms with a thin main()? AIt runs faster. BYou can unit-test pure transforms with injected inputs. The thin main only connects read/transform/write, and you integration-test it. CSpark requires it. DIt removes the need for any configuration.

6On EMR Serverless, what is the right habit for a large one-off benchmark job in a shared account? ARun it untagged and leave the application running for convenience. BRight-size the application, tag resources (e.g. project=de-mastery), and tear it down afterward. CHard-code credentials in the job so that it always works. DDisable AQE to save money.

7The test of a teammate asserts that the output columns are in the order [id, name, total]. A refactor renames nothing but reorders them to [id, total, name], and the test fails. What is the lesson? AThe refactor is wrong. BThe test is too strict. Column order is an incidental detail unless the task specifies it. CSpark changed the semantics. DYou must always assert column order.