DE Mastery GitHub

Track · STH · ~35 h

Streamhouse

Philosophy. Lambda architecture runs two pipelines, batch and streaming, which often disagree. Kappa runs one streaming pipeline, but Kafka storage is expensive to query historically. The lakehouse made object storage transactional, but only at batch latency. A streamhouse puts a stream-native storage layer on the lake, so one copy of the data serves streaming reads (changelogs) and batch/OLAP reads, usually with Flink as the engine. The term was coined by Ververica, and is now shared across several streaming vendors (Aiven, Confluent, Redpanda, StreamNative, Ververica), who have since formed a Streamhouse working group. When not to use it: if hourly freshness is fine, a batch lakehouse is simpler and cheaper.

Prerequisites: LKH, KFK, FLK. Lab profile: streamhouse.

Modules

  1. 01 Architecture history — Lambda → Kappa → lakehouse → streamhouse 3 h · 1 exercise
  2. 02 Apache Paimon — primary-key tables, merge engines, partial-update 6 h · 1 exercise
  3. 03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency 5 h · 1 exercise
  4. 04 Iceberg as streaming storage — commit interval vs small files 4 h · 1 exercise
  5. 05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join 6 h · 1 exercise
  6. 06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness 4 h · 1 exercise
  7. 07 Serving layer — OLAP on the streamhouse 4 h · 1 exercise
  8. 08 Latency tiers & cost — seconds, minutes, hours 3 h · 1 exercise