Skip to content

DE Mastery GitHub

Start

Curriculum
Roadmap
How to study with this course

Tracks

FND Foundations of distributed data

Overview
01 Why distributed?
02 Partitioning & shuffles
03 Replication & consensus
04 Time
05 Delivery semantics
06 Storage formats
07 Back-of-envelope estimation

SPK Apache Spark

Overview
01 Spark architecture
02 Lazy evaluation and the DAG
03 The DataFrame API
04 Spark SQL and data sources
05 Catalyst and the physical plan
06 Joins and aggregations at scale
07 Partitioning, bucketing, and the small-file problem
08 Memory, caching, and Adaptive Query Execution
09 UDFs and the Pandas API
10 Structured Streaming I — the micro-batch engine and exactly-once
11 Structured Streaming II — watermarks, state, and streaming SCD2
12 Testing and production Spark
BYO Build your own mini columnar query engine

LKH Lakehouse & open table formats

Overview
01 The Iceberg metadata tree
02 Commits & concurrency
03 Catalogs, branching & write-audit-publish
04 Row-level changes — SCD2, CoW & MoR
05 Table layout & maintenance
06 Delta Lake
07 Hudi & the format landscape
08 Engines on the lake
BYO Build your own mini table format

JVM Java for data engineers

Overview
01 Java for Python developers
02 Collections and streams
03 Build and dependencies
04 Serialization
05 Concurrency and the JVM
06 Testing and debugging

KFK Apache Kafka

Overview
01 The log — events, state, retention and replay
02 Topics, partitions, replication — leaders, ISR, acks, durability
03 Producers — batching, idempotence, ordering under retries
04 Consumers & groups — manual commits, at-least-once, no loss across crashes
05 Transactions & exactly-once — read-process-write, read_committed, fencing
06 Schema Registry & contracts
07 Kafka Connect & CDC
08 Kafka Streams
09 Event & topic design
10 Testing & observability (apps)
11 Cluster configuration & partition reassignment
12 Security — SASL/SCRAM authentication & ACL authorization
13 Deployment architecture — multi-region DR
14 Operating & troubleshooting a cluster
BYO Build your own mini Kafka

FLK Apache Flink

Overview
01 Philosophy & architecture
02 DataStream API
03 Time & windows
04 Flink SQL & dynamic tables
05 Joins — regular, interval, temporal (point-in-time) & lookup
06 State, checkpoints & exactly-once — surviving a TaskManager crash
07 Process functions — timers, keyed state & broadcast rules
08 Table API & PyFlink — one pipeline, two languages
09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER
10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution
11 Performance & state at scale — backpressure, parallelism, redistribution, skew
12 Deploy & operate — upgrade a stateful job via savepoint without losing state
BYO Build your own mini stream processor

STH Streamhouse

Overview
01 Architecture history — Lambda → Kappa → lakehouse → streamhouse
02 Apache Paimon — primary-key tables, merge engines, partial-update
03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency
04 Iceberg as streaming storage — commit interval vs small files
05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join
06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness
07 Serving layer — OLAP on the streamhouse
08 Latency tiers & cost — seconds, minutes, hours

Apache Flink · FLK-11

Performance & state at scale — backpressure, parallelism, redistribution, skew

6 h
requires FLK-02
requires FLK-06

From your world. Spark skew and shuffle tuning: one busy task holding up the stage, fixed by redistributing work across more parallel subtasks

Lesson Exercises 1 Quiz Interview drills Explain it back

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz FLK-11.

1What is backpressure in Flink? AA crash caused by too much data BFlow control. When an operator cannot keep up, its buffers fill and the full buffers spread upstream. The source slows down to the rate of the slowest operator CA way to drop late events DA checkpoint failure

Backpressure is the built-in flow control of Flink. Sustained backpressure shows the bottleneck operator.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/monitoring/back_pressure/

2A heavy, NON-keyed map operator runs at parallelism 1 and backpressures the source. Slots are free. What is the correct fix? ARaise the parallelism of the map and call rebalance() on the records before it, so the subtasks get records evenly BAdd a keyBy on a constant CDisable checkpointing DLower the source parallelism

More subtasks plus a redistributing edge (rebalance) spread the load evenly. keyBy on a constant would send everything to one subtask. This is the opposite of what you want.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/

3You raise the parallelism of a downstream operator to 4 but leave the edge FORWARD. Why does throughput not improve? AFORWARD is one-to-one at equal parallelism. It cannot feed 4 subtasks from 1, so it does not redistribute the work BFORWARD drops records CParallelism 4 is invalid DIt does improve

FORWARD needs equal upstream and downstream parallelism and does no redistribution. To feed more subtasks, you need a redistributing strategy (REBALANCE/RESCALE/HASH).

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/

4When does a higher parallelism NOT fix a bottleneck? AWhen the bottleneck is a hot key under keyBy (skew). That key still goes to one subtask BNever. Parallelism always helps COnly on bounded sources DOnly when checkpointing is on

keyBy sends a key to one subtask at any parallelism, so a hot key stays a bottleneck. The fix is to change the key (salting), not to add subtasks.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/

5Which ship strategy gives round-robin and perfectly even distribution to all downstream subtasks for a non-keyed operator? AFORWARD BREBALANCE CBROADCAST DGLOBAL

REBALANCE sends records round-robin across all downstream subtasks. It evens out a load without skew.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/

6How can you prove a throughput fix from the job graph alone, without timing? AYou cannot; only wall-clock proves it BThe parallelism of the bottleneck operator increased, and the edge that feeds it changed from FORWARD to a redistributing strategy (REBALANCE/RESCALE/HASH) CThe number of operators decreased DThe job name changed

A real fix shows in the structure: higher parallelism on the bottleneck and a redistributing edge that feeds it. This is deterministic and machine load does not change it.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/parallel/

7Why is RESCALE sometimes preferred over REBALANCE? ARESCALE drops duplicates BRESCALE sends records round-robin to a local subset of subtasks, so it avoids a full network shuffle when you scale by an integer factor CRESCALE guarantees ordering DThey are identical

RESCALE distributes records to a subset of downstream subtasks (local where possible). It costs less than the all-to-all distribution of REBALANCE when the parallelism ratio is an integer.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/operators/overview/

8Which are legitimate reasons why a higher parallelism may not help a backpressured job? Select all that apply. (select all that apply) AThe bottleneck is a hot key (skew), not low parallelism BA slow external sink is the real limit CHigher parallelism multiplies state and checkpoint cost DFORWARD always redistributes for you

Skew, a slow sink, and the cost of state and checkpoints all limit the benefit of raw parallelism. FORWARD does NOT redistribute, so D is false.

Source: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/dev/datastream/execution/parallel/

PreviousFLK-10 · CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution NextFLK-12 · Deploy & operate — upgrade a stateful job via savepoint without losing state

DE Mastery · source on GitHub · Found an error? Report it on GitHub