Row-level changes — SCD2, CoW & MoR

Iceberg data files are immutable. Iceberg implements row-level change in one of two ways. It rewrites the affected files (copy-on-write). Or it writes a separate delete file that logically removes rows, and the engine merges it at read time (merge-on-read).

Source: https://iceberg.apache.org/spec/#row-level-deletes

CoW rewrites whole files on change (write amplification). So reads see only clean data files. MoR writes a small delete file instead. So writes are cheap. But every read merges the deletes against the data until compaction folds them in.

Source: https://iceberg.apache.org/docs/latest/spark-writes/

MoR records the delete as a position-delete file (content=1). It leaves the data files (content=0) untouched. CoW, in contrast, would rewrite the data file and show only content=0.

Source: https://iceberg.apache.org/spec/#row-level-deletes

CoW produces no delete files. It rewrites the affected data file. So the set of data files changes (a new file replaces the old file), and content stays 0 throughout. The tests assert this measurable property. They do not assert timing.

Source: https://iceberg.apache.org/docs/latest/spark-writes/

SCD2 versions history. Iceberg closes the old row (valid_to is set, is_current=false). Then it inserts a new open row (valid_to NULL, is_current=true). If you overwrite the row in place, you destroy history. History is the whole purpose of Type 2.

Source: https://iceberg.apache.org/docs/latest/spark-writes/

A, B and C are real SCD2 defects. D is the correct behavior and not a bug. A changed attribute should produce a new current version.

Source: https://iceberg.apache.org/docs/latest/spark-writes/

The oracle recomputes the SCD2 result independently in DuckDB over the same batches. When two engines agree, this guards against a wrong hand-written expectation. This is the G6 validity gate.

Source: https://iceberg.apache.org/docs/latest/spark-writes/

MoR optimizes the write cost at the expense of the read cost. Copy-on-write is better for a dimension that you read constantly and write rarely. It keeps every read against clean data files.

Source: https://iceberg.apache.org/docs/latest/configuration/