How the tests are proven valid

A test is valid when it passes every correct solution and fails every wrong one. The most common failure modes in course test suites are:

1. Too weak: wrong code passes (for example, a test only checks the row count).
2. Too strict: correct but different code fails (for example, the test checks column order or a specific API call).
3. Flaky: results depend on timing, ordering or randomness, which is especially common in streaming.
4. Wrong: the expected output itself is wrong.

Every exercise must pass the validity gates below (de validate, run locally) before it ships. There is no hosted CI. Validate only the modules or stages you touched (de validate <ID> or --changed <ref>). Run the full cumulative de validate before a release.

Validity gates (checked by de validate for every exercise)

#	Gate	Catches
G1	Starter fails. The unmodified starter code fails at least one test	tests that pass vacuously
G2	Reference passes. solution/ passes all tests	broken tests, wrong expected data
G3	Alternate solutions pass. At least 2 references written differently (e.g., SQL vs DataFrame API, DataStream vs Flink SQL)	too-strict tests
G4	Mutants are killed. Every file in mutants/ (a plausible wrong solution) fails at least one test	too-weak tests
G5	No flakes. The full suite runs 20× with shuffled order (pytest-randomly), with zero flaky outcomes	timing/order dependence
G6	Oracle agreement. Expected outputs are generated by an independent engine, not hand-written	wrong expectations

G4: mutants are designed, not random

Each exercise ships hand-written mutants that encode the mistakes real engineers make. Examples:

Exercise	Mutants that must fail
SCD2 merge	overwrites instead of versioning; leaves valid_to open on old rows; off-by-one on effective dates
Broadcast join	inner join instead of left; broadcasts the large side; drops NULL keys
Windowed aggregation	processing time instead of event time; no watermark; late events silently dropped instead of sent to side output
Exactly-once sink	non-transactional producer; offsets committed before the write; non-idempotent upsert
Multi-company FX	converts at transaction date instead of the policy date; double-counts intercompany sales

We also run mutmut on reference solutions as a secondary check. Surviving automatic mutants are reviewed. Either a test is added, or the mutant is marked equivalent with a reason.

G6: oracles

• Batch: Spark output is compared with DuckDB running the same logic on the same seeded input. Two independent engines agreeing makes a wrong expected result very unlikely.
• Streaming: the key invariant is that when all input has arrived and watermarks have passed, the streaming result must equal the batch result over the same events. The test computes the batch answer with DuckDB, then asserts the streaming job's final materialized state matches it. Intermediate (retracted or updated) results are tested separately, against the changelog semantics.
• Property-based tests (Hypothesis): random event orders, duplicates, late events and NULLs. Properties include "deduplication is idempotent", "aggregations commute with input order within the watermark", and "sum(debits) == sum(credits)".

What tests assert (and what they don't)

• Assert behavior and invariants: output data, schema, row-level equality ignoring order (assertDataFrameEqual, checkRowOrder=False), and changelog contents.
• Assert plan properties, not timing. For performance exercises the test parses df.explain() / the Flink job graph. Examples: "a BroadcastHashJoin is present", "at most 1 Exchange", "no Python UDF in the plan". Wall-clock limits are used only as generous timeouts.
• Do not assert which API was called, column order (unless the task says so), or exact float values (use tolerances).

Streaming and fault tolerance tests

• Controlled time: tests use event time and synthetic watermarks, never sleep(). Flink tests use the MiniCluster and operator test harnesses. Spark tests use MemoryStream + processAllAvailable(). Kafka Streams tests use TopologyTestDriver.
• Fault injection: exactly-once exercises run against real containers (Testcontainers). The test kills a TaskManager, broker or Spark executor mid-run (and uses Toxiproxy for network faults), then checks ledger invariants: no duplicate IDs, no gaps, balanced totals.
• Determinism: fixed seeds, fixed partition counts, and pinned versions (labs/versions.lock).

BYO challenge tests

Stage tests are black-box: a real Kafka client (e.g., kcat, confluent-kafka), a real Parquet/Avro reader, or a socket protocol talks to your implementation. This tests real behavior and lets you use any language. Each stage also has a regression suite: earlier stages must keep passing.

Quizzes

• Each question cites the official documentation section it is based on.
• Distractors are written from real misconceptions (the same idea as mutants).
• Questions with a miss rate above 70% are flagged for review, since the question may be ambiguous.

When you think a test is wrong

Open an issue on the repo with your solution, the failing test and your argument. A disputed test is re-checked against G2/G3/G6. If your solution is correct, it becomes a new alternate solution (G3) and the test is fixed.

Who checks the checker

• Tests and reference solutions are written by one author and audited by a separate reviewer who sees only the task statement, then tries to write passing-but-wrong code. Any such code it finds becomes a new mutant.
• Each test is labeled with the official doc page it is based on, so its behavior can be traced to the source (e.g., Flink's watermark idleness semantics) rather than to someone's memory.