Apache Spark · SPK-11
Structured Streaming II — watermarks, state, and streaming SCD2
- 7 h
- requires SPK-10
From your world. SCD2 dimension snapshots and late-arriving facts
- Event-time windowed counts with a watermark into IcebergSPK-11/01-watermark-window
- Bounded streaming deduplication with a watermarkSPK-11/02-dedup-within-watermark
- Streaming SCD2 upsert into Iceberg with foreachBatch + MERGE INTOSPK-11/03-streaming-scd2-merge
Run it locally
uv run de start SPK-11
uv run de test