Foundations of distributed data · FND-04
Time
- 3 h
- requires FND-01
From your world. Late-arriving facts and dimensions in Kimball; how long to hold a window open before you trust the number
Explain to a Kimball modeler who reprocesses "the last few days" every night:
"What is a watermark, why must it only move forward, and how is
allowedLatenessthe same decision as my dbt lookback window?"
Rubric (self-check — aim for 5/6)
- Distinguishes event time from processing time. Says why they diverge (network, retries, buffering).
- Defines a watermark as a completeness claim: "seen all events with event time ≤ W".
- States the monotonic / non-decreasing rule and why (closed windows must not re-open).
- Gives the bounded-out-of-orderness formula
W = maxEventTime − allowedLateness. - Classifies events as on-time / late-accepted / dropped relative to W and the grace window. Says that late events in auditable data go to a side output, not to a silent drop.
- Maps
allowedLatenessto the dbt lookback window / Kimball late-arriving facts.