Catalogs, branching & write-audit-publish

The catalog resolves db.table to the current metadata.json. It performs the atomic pointer swap that is a commit. It also governs access (Glue/Lake Formation, Unity, Polaris, Nessie, Lakekeeper). The data files are portable across catalogs. The catalog is the control plane.

Source: https://iceberg.apache.org/docs/latest/branching/

A tag names a fixed snapshot (a point in time). A branch is a movable reference, like a git branch. You can write to a branch without an effect on readers of main.

Source: https://iceberg.apache.org/spec/#snapshot-references

When you write to a branch, the new data is isolated from main. Readers of main continue to see the last published state while you validate the branch. Only a passing audit publishes the data to main.

Source: https://iceberg.apache.org/docs/latest/branching/

The goal is to validate the candidate data before it reaches main. So the checks must read the snapshot of the AUDIT branch. Main does not contain the new data yet. If you audit main, you make a classic WAP mistake. It passes bad data straight through.

Source: https://iceberg.apache.org/docs/latest/branching/

fast_forward advances main to the tip of the audit branch. This publishes the staged snapshots. It is the 'publish' step of WAP. The alternative is cherrypick_snapshot. Use it when you want to apply a single snapshot of the branch onto a main that has moved.

Source: https://iceberg.apache.org/docs/latest/spark-procedures/

A control works only if the gate is enforced before the data is visible. Suppose you publish before the check. Or suppose you check a snapshot that does not contain the candidate data. Then bad rows reach production. The audited-publish control exists to prevent exactly this failure.

Source: https://iceberg.apache.org/docs/latest/branching/

Branching adds snapshots and maintenance overhead. It is worth it when a bad publish is costly or when someone must approve it. For cheap, low-risk appends, the overhead is not justified.

Source: https://iceberg.apache.org/docs/latest/branching/