DE Mastery GitHub

Track · LKH · ~35 h

Lakehouse & open table formats

Philosophy. A table format turns a pile of files in object storage into a database table. It provides ACID commits, snapshots, schema evolution and partition evolution, and keeps the data open to any engine. You already use Iceberg. This track teaches its internals, how to operate it, and the alternatives, so that "Delta shop" or "Hudi shop" in a job posting doesn't scare you. When not to use it: small operational data (use Postgres), sub-second point lookups (use a KV store), or a team that is fully inside one warehouse and has no need for openness.

Lab profile: core (Spark + Iceberg REST + MinIO + Trino).

Modules

  1. 01 The Iceberg metadata tree 4 h · 2 exercises
  2. 02 Commits & concurrency 4 h · 1 exercise
  3. 03 Catalogs, branching & write-audit-publish 4 h · 1 exercise
  4. 04 Row-level changes — SCD2, CoW & MoR 5 h · 2 exercises
  5. 05 Table layout & maintenance 5 h · 1 exercise
  6. 06 Delta Lake 4 h · 1 exercise
  7. 07 Hudi & the format landscape 3 h · 1 exercise
  8. 08 Engines on the lake 6 h · 1 exercise
  9. BYOBuild your own mini table format 8 h · 7 stages