Skip to content

DE Mastery GitHub

Start

Curriculum
Roadmap
How to study with this course

Tracks

FND Foundations of distributed data

Overview
01 Why distributed?
02 Partitioning & shuffles
03 Replication & consensus
04 Time
05 Delivery semantics
06 Storage formats
07 Back-of-envelope estimation

SPK Apache Spark

Overview
01 Spark architecture
02 Lazy evaluation and the DAG
03 The DataFrame API
04 Spark SQL and data sources
05 Catalyst and the physical plan
06 Joins and aggregations at scale
07 Partitioning, bucketing, and the small-file problem
08 Memory, caching, and Adaptive Query Execution
09 UDFs and the Pandas API
10 Structured Streaming I — the micro-batch engine and exactly-once
11 Structured Streaming II — watermarks, state, and streaming SCD2
12 Testing and production Spark
BYO Build your own mini columnar query engine

LKH Lakehouse & open table formats

Overview
01 The Iceberg metadata tree
02 Commits & concurrency
03 Catalogs, branching & write-audit-publish
04 Row-level changes — SCD2, CoW & MoR
05 Table layout & maintenance
06 Delta Lake
07 Hudi & the format landscape
08 Engines on the lake
BYO Build your own mini table format

JVM Java for data engineers

Overview
01 Java for Python developers
02 Collections and streams
03 Build and dependencies
04 Serialization
05 Concurrency and the JVM
06 Testing and debugging

KFK Apache Kafka

Overview
01 The log — events, state, retention and replay
02 Topics, partitions, replication — leaders, ISR, acks, durability
03 Producers — batching, idempotence, ordering under retries
04 Consumers & groups — manual commits, at-least-once, no loss across crashes
05 Transactions & exactly-once — read-process-write, read_committed, fencing
06 Schema Registry & contracts
07 Kafka Connect & CDC
08 Kafka Streams
09 Event & topic design
10 Testing & observability (apps)
11 Cluster configuration & partition reassignment
12 Security — SASL/SCRAM authentication & ACL authorization
13 Deployment architecture — multi-region DR
14 Operating & troubleshooting a cluster
BYO Build your own mini Kafka

FLK Apache Flink

Overview
01 Philosophy & architecture
02 DataStream API
03 Time & windows
04 Flink SQL & dynamic tables
05 Joins — regular, interval, temporal (point-in-time) & lookup
06 State, checkpoints & exactly-once — surviving a TaskManager crash
07 Process functions — timers, keyed state & broadcast rules
08 Table API & PyFlink — one pipeline, two languages
09 Streaming patterns — dedup, top-N, split/merge, MATCH_RECOGNIZE, OVER
10 CDC & lake sinks — Debezium changelog to Iceberg, commit on checkpoint, schema evolution
11 Performance & state at scale — backpressure, parallelism, redistribution, skew
12 Deploy & operate — upgrade a stateful job via savepoint without losing state
BYO Build your own mini stream processor

STH Streamhouse

Overview
01 Architecture history — Lambda → Kappa → lakehouse → streamhouse
02 Apache Paimon — primary-key tables, merge engines, partial-update
03 Apache Fluss — PK tables, lake tiering, streaming vs batch consistency
04 Iceberg as streaming storage — commit interval vs small files
05 Streaming dimensional modeling — SCD2 dim + fct with temporal (as-of) join
06 Streaming data quality — in-flight DQ, quarantine / DLQ, completeness
07 Serving layer — OLAP on the streamhouse
08 Latency tiers & cost — seconds, minutes, hours

Java for data engineers · JVM-06

Testing and debugging

3 h
requires JVM-01
requires JVM-02

From your world. dbt tests + a Spark pipeline integration test, but as JUnit/AssertJ and reading JVM stack traces

Lesson Exercises 1 Quiz Interview drills Explain it back

Practice quiz with instant feedback. For a scored run that counts toward your local progress, use uv run de quiz JVM-06.

1In a Java stack trace with nested Caused by: sections, where is the root cause? AThe very first line (top exception). BThe last Caused by: section (the deepest cause), then the first frame in your own package. CThe last line of the whole trace regardless of content. DAny java.base frame.

Java prints the outermost exception first and nests causes under Caused by:. The deepest Caused by is the root cause. Then the first stack frame in your package (not JDK/library frames) is usually where the bug is.

Source: https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Throwable.html

2Which AssertJ assertion correctly tests that a method throws IllegalArgumentException? AassertThat(method()).isInstanceOf(IllegalArgumentException.class) BassertThatThrownBy(() -> method()).isInstanceOf(IllegalArgumentException.class) CassertThrows(method()) DassertThat(() -> method()).throwsException()

assertThatThrownBy takes a lambda, calls it, captures the thrown Throwable, and lets you assert its type and message. If you call method() directly in assertThat, the exception happens before the assertion runs.

Source: https://assertj.github.io/doc/#assertj-core-assertions-guide

3Why compare doubles with isCloseTo(v, within(1e-9)) rather than isEqualTo(v)? AisEqualTo is deprecated. BFloating-point arithmetic has rounding error, so exact equality is brittle. A tolerance makes the test stable and correct. Cwithin is faster. DDoubles cannot be compared at all.

Because of IEEE-754 rounding, computations that are mathematically equal can differ in the last bits. Assert with a tolerance (within) for floating point, as the platform's testing rules require.

Source: https://assertj.github.io/doc/#assertj-core-assertions-guide

4What does Testcontainers provide for a JVM integration test, and what does it require? AA mock broker in-process. It requires nothing. BTemporary real dependencies (e.g. a Kafka/Postgres container) that start and stop for each test. It requires Docker. CA way to skip all tests. DA replacement for JUnit.

Testcontainers starts real services in Docker containers for the test and removes them afterward. These integration tests are close to production, but Docker must be available. This is why grading has a no-Docker native-broker path.

Source: https://java.testcontainers.org/modules/kafka/

5Per this platform's rules, how must a graded Kafka exercise work without Docker? AIt cannot. Kafka needs Docker. BIt uses the native kafka_broker (a real Kafka JVM from the tarball) or TopologyTestDriver. Any Testcontainers/lab layer is an EXTRA that skips when Docker/the lab is down. CIt mocks Kafka entirely. DIt only runs on CI.

The native broker (no container) or an in-process TopologyTestDriver provides the no-Docker graded path. A skipped test cannot fail G1, so the container/lab layer is additional only.

Source: https://junit.org/junit5/docs/current/user-guide/

6Why should an integration test poll for the expected records with a deadline instead of calling Thread.sleep(2000)? Asleep is illegal in Java. BA fixed sleep is both flaky (too short) and slow (too long). Polling with a deadline is deterministic and fast. It finishes as soon as the data arrives. CPolling uses less memory. DThere is no difference.

Fixed sleeps make tests either flaky or needlessly slow. Poll (e.g. consumer.poll in a loop) until you have the expected records or reach a timeout. This is deterministic and quick.

Source: https://junit.org/junit5/docs/current/user-guide/

7A trace ends with ... 3 more. What does that mean? AThree records were dropped. BThe remaining 3 frames are identical to the frames of the enclosing trace. The trace leaves them out to be short. CThree more exceptions were thrown. DThe JVM ran out of stack.

'... N more' means that frames are left out. Those N frames match the enclosing stack trace. It is not a separate failure path.

Source: https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Throwable.html

PreviousJVM-05 · Concurrency and the JVM NextKFK-01 · The log — events, state, retention and replay

DE Mastery · source on GitHub · Found an error? Report it on GitHub