Skip to content

DE Mastery GitHub

Start

Curriculum
Roadmap
How to study with this course

Tracks

FND Foundations of distributed data

Overview
01 Why distributed?
02 Partitioning & shuffles
03 Replication & consensus
04 Time
05 Delivery semantics
06 Storage formats
07 Back-of-envelope estimation

SPK Apache Spark

Overview
01 Spark architecture
02 Lazy evaluation and the DAG
03 The DataFrame API
04 Spark SQL and data sources
05 Catalyst and the physical plan
06 Joins and aggregations at scale
07 Partitioning, bucketing, and the small-file problem
08 Memory, caching, and Adaptive Query Execution
09 UDFs and the Pandas API
10 Structured Streaming I — the micro-batch engine and exactly-once
11 Structured Streaming II — watermarks, state, and streaming SCD2
12 Testing and production Spark
BYO Build your own mini columnar query engine

LKH Lakehouse & open table formats

Overview
01 The Iceberg metadata tree
02 Commits & concurrency
03 Catalogs, branching & write-audit-publish
04 Row-level changes — SCD2, CoW & MoR
05 Table layout & maintenance
06 Delta Lake
07 Hudi & the format landscape
08 Engines on the lake
BYO Build your own mini table format

JVM Java for data engineers

Overview
01 Java for Python developers
02 Collections and streams
03 Build and dependencies
04 Serialization
05 Concurrency and the JVM
06 Testing and debugging

KFK Apache Kafka

Overview
01 The log — events, state, retention and replay
02 Topics, partitions, replication — leaders, ISR, acks, durability
03 Producers — batching, idempotence, ordering under retries
04 Consumers & groups — manual commits, at-least-once, no loss across crashes
05 Transactions & exactly-once — read-process-write, read_committed, fencing
06 Schema Registry & contracts
07 Kafka Connect & CDC
08 Kafka Streams
09 Event & topic design
10 Testing & observability (apps)
11 Cluster configuration & partition reassignment
12 Security — SASL/SCRAM authentication & ACL authorization
13 Deployment architecture — multi-region DR
14 Operating & troubleshooting a cluster
BYO Build your own mini Kafka

FLK Apache Flink

Overview
01 Philosophy & architecture
02 DataStream API
03 Time & windows
04 Flink SQL & dynamic tables
05 Joins — regular, interval, temporal (point-in-time) & lookup
06 State, checkpoints & exactly-once — surviving a TaskManager crash
07 Process functions — timers, keyed state & broadcast rules
08 Table API & PyFlink — one pipeline, two languages
09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER
10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution
11 Performance & state at scale — backpressure, parallelism, redistribution, skew
12 Deploy & operate — upgrade a stateful job via savepoint without losing state
BYO Build your own mini stream processor

STH Streamhouse

Overview
01 Architecture history — Lambda → Kappa → lakehouse → streamhouse
02 Apache Paimon — primary-key tables, merge engines, partial-update
03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency
04 Iceberg as streaming storage — commit interval vs small files
05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join
06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness
07 Serving layer — OLAP on the streamhouse
08 Latency tiers & cost — seconds, minutes, hours

Foundations of distributed data · FND-06

Storage formats

3 h
requires FND-01

From your world. Why Athena charges by scanned bytes; row-group pruning is why a WHERE on a sorted column is cheap

Lesson Exercises 1 Quiz Interview drills Explain it back

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz FND-06.

1Why can a columnar format read far fewer bytes than a row format for an analytical query? AIt stores each column contiguously, so a query reads only the columns it references BIt compresses the whole file into one block CIt stores rows in sorted order always DIt avoids storing a schema

A columnar layout lets the engine read only the bytes of the columns that the query references (and prune row groups). It does not read every field of every row.

Source: https://parquet.apache.org/docs/file-format/

2For WHERE x > 900, which row groups can Parquet skip without reading data pages? AAny row group whose stored max(x) <= 900 BAny row group whose min(x) > 900 COnly the first row group DNone — Parquet always reads all pages

If max(x) <= 900 in a row group, no row in it can satisfy x > 900. So the engine prunes the group with the footer stats alone.

Source: https://arrow.apache.org/docs/python/parquet.html

3Row-group pruning is most effective when the filtered column is: ASorted or clustered, so each group has a tight, non-overlapping [min,max] BRandomly ordered CA complex nested type with no stats DThe highest-cardinality column, unsorted

Sorting or clustering makes the range of each group tight and disjoint, so a predicate overlaps few groups. Random order makes the range of every group span the domain. Then nothing is pruned.

Source: https://iceberg.apache.org/spec/

4Athena charges by bytes scanned. The biggest single cost lever is usually: AStore Parquet and SELECT only needed columns (column pruning) BIncrease the Athena workgroup memory CUse SELECT * on JSON DDisable compression

Columnar storage plus projection of only the needed columns cuts the scanned bytes the most. Partitioning and sort order add row-group and file pruning on top.

Source: https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html

5Which format is the better fit for streaming writes of one record at a time (for example Kafka)? AAvro (a compact row format) BParquet CORC DCSV with no schema

Avro is a row format with a schema. It suits writes of one record at a time. Columnar formats (Parquet/ORC) are for batched analytical scans.

Source: https://parquet.apache.org/docs/file-format/

6What do the footer statistics of a Parquet column chunk typically include? (select all) (select all that apply) Amin Bmax Cnull_count DThe full list of row ids

The stats of each column chunk include min, max, and null_count (and often distinct_count). They do not store a full row-id list.

Source: https://parquet.apache.org/docs/file-format/

7How does Iceberg prune whole files before opening them? AIt keeps min/max column metrics for each file in manifests. It skips files whose range cannot match the predicate BIt scans every file's footer at query time CIt stores all data in a single file DIt relies on the OS page cache

Iceberg manifests carry column metrics (min/max/null counts) for each data file. So the planner prunes non-matching files before it reads them. Sort order makes those ranges tighter.

Source: https://iceberg.apache.org/spec/

PreviousFND-05 · Delivery semantics NextFND-07 · Back-of-envelope estimation

DE Mastery · source on GitHub · Found an error? Report it on GitHub