Streamhouse · STH-07
Serving layer — OLAP on the streamhouse
From your world. Athena/Redshift serving marts: a dashboard reads a small pre-aggregated table, not the raw fact
Prompt (Feynman). A teammate says: "The dashboard is slow, so just point Trino straight at the raw Paimon fact table and add more CPU." Explain in your own words why a serving layer on a streamhouse reads a small pre-aggregated table instead. Explain how you read the merged state of a primary-key table (not its changelog). Explain how you would prove that the OLAP answer is correct. Use the revenue-per-company-per-day mart as your running example.
Rubric — your explanation should hit these (5–7)
- Two read modes. The same lake table serves a streaming (changelog) read and a batch/OLAP (snapshot) read. The serving layer is the batch/OLAP one, for interactive queries.
- Merged state, not changelog. For a PK / merge-engine table, you read the latest value per key with merges applied (pypaimon / pyiceberg batch snapshot). You never iterate the unbounded changelog.
- Pre-aggregation. The dashboard reads a small rollup (revenue per company per day), not the raw fact. The rows that it reads drop by orders of magnitude.
- Pruning & projection. A predicate (one company, a date range) lets a columnar engine skip row groups and columns. You can see this in the query plan.
- The plan, not the timing, is the proof. You verify that the dashboard query reads the
rollup (via
EXPLAIN). Wall-clock time is only a generous guardrail (warm cache, median of N). - Correctness invariant. The OLAP answer over the merged snapshot must equal a batch rebuild of the same events (G6). A disagreement means a stale snapshot or double-counted un-merged versions.
- When not to. Drill-to-detail on one key is a point lookup, not a rollup. A pipeline consumer should read the changelog, not an OLAP snapshot.