Delta Lake

Delta rebuilds the state of a table when it replays the ordered commits in _delta_log/ (add/remove actions). Iceberg instead keeps a catalog pointer to a single metadata.json. Both formats are append-only, with atomic optimistic-concurrency commits. The shape of the metadata is different.

Source: https://github.com/delta-io/delta/blob/master/PROTOCOL.md

A replay of thousands of JSON commits is slow. So Delta periodically writes a checkpoint Parquet file. It captures the full live-file set at a version. Readers load the latest checkpoint and replay only the commits after it. This is the scaling mechanism of Delta. It is similar to Iceberg manifest lists.

Source: https://docs.delta.io/latest/delta-utility.html#history

SCD2 preserves history. Close the current version when you set valid_to. Then insert a new current version (valid_to NULL). If you overwrite the attribute in place, you get SCD Type 1, and you destroy history. This is the classic mistake. If you delete the old row, you also lose history.

Source: https://docs.delta.io/latest/delta-update.html#upsert-into-a-table-using-merge

An UPDATE emits both the preimage (the row before) and the postimage (the row after). So a consumer sees the exact before and after. Inserts emit 'insert'. Deletes emit 'delete'. Each change row also carries _commit_version.

Source: https://docs.delta.io/latest/delta-change-data-feed.html

You must enable CDF (delta.enableChangeDataFeed=true) before the changes that you want to read. It adds write cost and change files. The feed does not have the changes that were written before you turned it on.

Source: https://docs.delta.io/latest/delta-change-data-feed.html

Liquid clustering replaces static hive partitioning plus ZORDER with a CLUSTER BY that Delta maintains incrementally. It improves data skipping, and it avoids the problems of fixed partition columns. It plays the role that Iceberg fills with sort orders and hidden partitioning.

Source: https://docs.delta.io/latest/delta-clustering.html

UniForm (Universal Format) writes Iceberg-compatible (and Hudi-compatible) metadata next to the Delta log. Engines that speak those formats can then read the same underlying data. This is interoperability without a copy of the data.

Source: https://docs.delta.io/latest/delta-uniform.html