Foundations of distributed data · FND-01
Why distributed?
- 2 h
From your world. Why Redshift has a leader node and compute nodes; when one big box + DuckDB beats a cluster
Explain to a colleague who knows SQL but not distributed systems:
"Why does adding more machines eventually stop making my query faster, and how do I decide whether to use one big box or a cluster at all?"
Talk for about 5 minutes. Use a Redshift leader/compute analogy.
Rubric (self-check — aim for 5/6)
- States the two limits that force distribution: capacity and time/deadline.
- Explains Amdahl's law in words: the serial fraction caps the speedup at
1/(1-p). - Names a concrete serial bottleneck (final sort,
collect()to driver, leader-node merge). - Contrasts scale up vs scale out and names the price of scale-out (the shuffle / network).
- Gives a bytes-based reason why a single box with DuckDB can beat a cluster (columnar pruning → fewer bytes read).
- States at least one concrete "do NOT distribute" condition (fits in RAM, serial-bound, or latency-sensitive small query).