Skip to content

DE Mastery GitHub

Start

Curriculum
Roadmap
How to study with this course

Tracks

FND Foundations of distributed data

Overview
01 Why distributed?
02 Partitioning & shuffles
03 Replication & consensus
04 Time
05 Delivery semantics
06 Storage formats
07 Back-of-envelope estimation

SPK Apache Spark

Overview
01 Spark architecture
02 Lazy evaluation and the DAG
03 The DataFrame API
04 Spark SQL and data sources
05 Catalyst and the physical plan
06 Joins and aggregations at scale
07 Partitioning, bucketing, and the small-file problem
08 Memory, caching, and Adaptive Query Execution
09 UDFs and the Pandas API
10 Structured Streaming I — the micro-batch engine and exactly-once
11 Structured Streaming II — watermarks, state, and streaming SCD2
12 Testing and production Spark
BYO Build your own mini columnar query engine

LKH Lakehouse & open table formats

Overview
01 The Iceberg metadata tree
02 Commits & concurrency
03 Catalogs, branching & write-audit-publish
04 Row-level changes — SCD2, CoW & MoR
05 Table layout & maintenance
06 Delta Lake
07 Hudi & the format landscape
08 Engines on the lake
BYO Build your own mini table format

JVM Java for data engineers

Overview
01 Java for Python developers
02 Collections and streams
03 Build and dependencies
04 Serialization
05 Concurrency and the JVM
06 Testing and debugging

KFK Apache Kafka

Overview
01 The log — events, state, retention and replay
02 Topics, partitions, replication — leaders, ISR, acks, durability
03 Producers — batching, idempotence, ordering under retries
04 Consumers & groups — manual commits, at-least-once, no loss across crashes
05 Transactions & exactly-once — read-process-write, read_committed, fencing
06 Schema Registry & contracts
07 Kafka Connect & CDC
08 Kafka Streams
09 Event & topic design
10 Testing & observability (apps)
11 Cluster configuration & partition reassignment
12 Security — SASL/SCRAM authentication & ACL authorization
13 Deployment architecture — multi-region DR
14 Operating & troubleshooting a cluster
BYO Build your own mini Kafka

FLK Apache Flink

Overview
01 Philosophy & architecture
02 DataStream API
03 Time & windows
04 Flink SQL & dynamic tables
05 Joins — regular, interval, temporal (point-in-time) & lookup
06 State, checkpoints & exactly-once — surviving a TaskManager crash
07 Process functions — timers, keyed state & broadcast rules
08 Table API & PyFlink — one pipeline, two languages
09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER
10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution
11 Performance & state at scale — backpressure, parallelism, redistribution, skew
12 Deploy & operate — upgrade a stateful job via savepoint without losing state
BYO Build your own mini stream processor

STH Streamhouse

Overview
01 Architecture history — Lambda → Kappa → lakehouse → streamhouse
02 Apache Paimon — primary-key tables, merge engines, partial-update
03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency
04 Iceberg as streaming storage — commit interval vs small files
05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join
06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness
07 Serving layer — OLAP on the streamhouse
08 Latency tiers & cost — seconds, minutes, hours

Foundations of distributed data · FND-05

Delivery semantics

3 h
requires FND-01

From your world. Re-running a dbt model safely; SOX 'completeness and accuracy'; idempotent MERGE on a natural key

Lesson Exercises 1 Quiz Interview drills Explain it back

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz FND-05.

1'Exactly-once' in practice is usually achieved by: AAt-least-once delivery plus idempotent or transactional processing at the sink BA network that never duplicates messages and never drops messages CSending every message one time and hoping DAt-most-once delivery with retries disabled

You rarely get exactly-once on the wire. You get effectively-once when you deliver at-least-once and make the effect of the sink idempotent or transactional.

Source: https://kafka.apache.org/documentation/#semantics

2What is the defining property of an idempotent write? AApplying the same event twice has the same effect as applying it once BIt is always faster than a transaction CIt never requires a key DIt guarantees messages arrive in order

Idempotence means f(f(x)) = f(x). If you apply the same keyed event again, the result does not change.

Source: https://kafka.apache.org/documentation/#semantics

3Why is a UUID generated at send time a BAD dedup key? AIt changes on each retry, so duplicates get different keys and the system does not recognize them as duplicates BUUIDs are too long to index CIt is always identical across events DIt sorts incorrectly

A dedup key must be STABLE across retries. A fresh UUID for each send makes the same logical event look new every time. This defeats dedup.

Source: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

4Which maps to the SOX 'completeness' control (no acknowledged transaction lost)? AUse at-least-once (never at-most-once) at ingestion BUse at-most-once to avoid duplicates CDisable retries DDedup on wall-clock time

Completeness forbids loss, so you must never drop an acknowledged event. This means at-least-once. Idempotent keyed application then handles accuracy (no double counting).

Source: https://research.google/pubs/pub43864/

5The two generals problem implies that over an unreliable channel: ANo finite exchange of acks can make both sides certain, so a sender must choose between resending (duplicates) and giving up (loss) BMessages can always be delivered exactly once with enough acks CAcknowledgements are unnecessary DOrdering is always preserved

This impossibility result is exactly why we pick a delivery guarantee. We then resolve duplicates or loss with idempotency or transactions.

Source: https://en.wikipedia.org/wiki/Two_Generals%27_Problem

6A good dedup key must be: (select all) (select all that apply) AStable across retries of the same logical event BUnique per distinct logical event CRandomly regenerated on each send DMatched to the event's true grain (not too coarse)

The key must be stable, unique, and at the right grain. If you regenerate it for each send (C), stability breaks and dedup fails.

Source: https://kafka.apache.org/documentation/#semantics

7An incremental dbt model with a unique_key and MERGE strategy, re-run on overlapping data, is: AIdempotent — the target ends up identical (f(f(x)) = f(x)) BAt-most-once — it may lose rows CGuaranteed to double-count DNon-deterministic

MERGE on a unique_key upserts, so reprocessing overlapping data gives the same target. This is the batch form of idempotent exactly-once.

Source: https://kafka.apache.org/documentation/#semantics

PreviousFND-04 · Time NextFND-06 · Storage formats

DE Mastery · source on GitHub · Found an error? Report it on GitHub