Skip to content

DE Mastery GitHub

Start

Curriculum
Roadmap
How to study with this course

Tracks

FND Foundations of distributed data

Overview
01 Why distributed?
02 Partitioning & shuffles
03 Replication & consensus
04 Time
05 Delivery semantics
06 Storage formats
07 Back-of-envelope estimation

SPK Apache Spark

Overview
01 Spark architecture
02 Lazy evaluation and the DAG
03 The DataFrame API
04 Spark SQL and data sources
05 Catalyst and the physical plan
06 Joins and aggregations at scale
07 Partitioning, bucketing, and the small-file problem
08 Memory, caching, and Adaptive Query Execution
09 UDFs and the Pandas API
10 Structured Streaming I — the micro-batch engine and exactly-once
11 Structured Streaming II — watermarks, state, and streaming SCD2
12 Testing and production Spark
BYO Build your own mini columnar query engine

LKH Lakehouse & open table formats

Overview
01 The Iceberg metadata tree
02 Commits & concurrency
03 Catalogs, branching & write-audit-publish
04 Row-level changes — SCD2, CoW & MoR
05 Table layout & maintenance
06 Delta Lake
07 Hudi & the format landscape
08 Engines on the lake
BYO Build your own mini table format

JVM Java for data engineers

Overview
01 Java for Python developers
02 Collections and streams
03 Build and dependencies
04 Serialization
05 Concurrency and the JVM
06 Testing and debugging

KFK Apache Kafka

Overview
01 The log — events, state, retention and replay
02 Topics, partitions, replication — leaders, ISR, acks, durability
03 Producers — batching, idempotence, ordering under retries
04 Consumers & groups — manual commits, at-least-once, no loss across crashes
05 Transactions & exactly-once — read-process-write, read_committed, fencing
06 Schema Registry & contracts
07 Kafka Connect & CDC
08 Kafka Streams
09 Event & topic design
10 Testing & observability (apps)
11 Cluster configuration & partition reassignment
12 Security — SASL/SCRAM authentication & ACL authorization
13 Deployment architecture — multi-region DR
14 Operating & troubleshooting a cluster
BYO Build your own mini Kafka

FLK Apache Flink

Overview
01 Philosophy & architecture
02 DataStream API
03 Time & windows
04 Flink SQL & dynamic tables
05 Joins — regular, interval, temporal (point-in-time) & lookup
06 State, checkpoints & exactly-once — surviving a TaskManager crash
07 Process functions — timers, keyed state & broadcast rules
08 Table API & PyFlink — one pipeline, two languages
09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER
10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution
11 Performance & state at scale — backpressure, parallelism, redistribution, skew
12 Deploy & operate — upgrade a stateful job via savepoint without losing state
BYO Build your own mini stream processor

STH Streamhouse

Overview
01 Architecture history — Lambda → Kappa → lakehouse → streamhouse
02 Apache Paimon — primary-key tables, merge engines, partial-update
03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency
04 Iceberg as streaming storage — commit interval vs small files
05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join
06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness
07 Serving layer — OLAP on the streamhouse
08 Latency tiers & cost — seconds, minutes, hours

Foundations of distributed data · FND-03

Replication & consensus

3 h
requires FND-01

From your world. Why Athena never 'loses' an S3 object; durability from replication and quorums vs. a single node

Lesson Exercises 1 Quiz Interview drills Explain it back

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz FND-03.

1With N=5 replicas, which quorum setting guarantees that every read sees the latest acknowledged write? AW=3, R=3 (W + R > N) BW=1, R=1 CW=2, R=2 DW=5, R=0

W + R > N (3 + 3 = 6 > 5) forces the read set to overlap the write set on at least one replica. That replica has the newest value.

Source: https://raft.github.io/raft.pdf

2In Raft, why can there be at most one leader per term? AA node votes at most once per term and a winner needs a majority. So two winners would need an overlapping voter to vote twice BTerms are randomly assigned so collisions are impossible CThe leader with the lowest node id always wins DFollowers cannot become candidates

One vote per term and a majority to win mean this: two leaders in the same term would need two majorities. The two majorities must share a voter who would have voted twice. This is a contradiction.

Source: https://raft.github.io/raft.pdf

3Kafka: acks=all with min.insync.replicas=1 on a partition with 3 replicas. What durability do you really have? AWeak — a single broker failure after the ack can lose acknowledged data BMajority quorum durability CThe same as acks=0 DExactly-once by construction

acks=all waits for the current ISR. But if min.insync.replicas=1, the ISR can be just the leader. If the leader is lost, the data is lost. You need M = floor(N/2)+1 for a majority.

Source: https://kafka.apache.org/documentation/#replication

4Why does Athena effectively never 'lose' an S3 object that you successfully PUT? AS3 replicates the object across devices/facilities. It acks the PUT only when the object is durably stored on enough of them (a quorum write) BAthena keeps a local cache of every object CS3 stores a single copy on very reliable hardware DGlue scans and rebuilds lost objects

S3 durability comes from redundant storage with quorum-acknowledged writes. The ack means that the object is durably replicated.

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html

5A Raft node receives a RequestVote for term T. It should grant its vote only if: AIt has not already voted in term T and the candidate's log is at least as up to date as its own BThe candidate has the highest node id CIt has already voted for someone else this term DThe term T is lower than its current term

A node votes at most once per term. It votes only for a candidate whose log is at least as current. This preserves leader completeness and single-leader safety.

Source: https://raft.github.io/raft.pdf

6Which statements about majority quorums are true? (select all) (select all that apply) AAny two majorities of the same set share at least one member BA majority of 4 nodes is 3 CA 2-of-4 vote is a valid majority DMajority overlap is what prevents two conflicting decisions

Majority = floor(N/2)+1 (so 3 of 4). Any two majorities intersect, and this overlap blocks split-brain. 2-of-4 is not a strict majority.

Source: https://raft.github.io/raft.pdf

PreviousFND-02 · Partitioning & shuffles NextFND-04 · Time

DE Mastery · source on GitHub · Found an error? Report it on GitHub