Apache Paimon — primary-key tables, merge engines, partial-update

Paimon stores PK-table data in an LSM tree. This is a set of small sorted files that the system compacts from time to time, like the storage engine of a streaming database. This makes cheap in-place upserts and a streaming changelog possible on object storage.

Source: https://paimon.apache.org/docs/master/concepts/overview/

partial-update keeps the latest non-null value per column. So partial rows from different writers combine into one complete row. This is exactly an MDM golden record.

Source: https://paimon.apache.org/docs/master/concepts/primary-key-table/merge-engine/

deduplicate keeps the last whole record (last-writer-wins). Each source sends NULLs for the columns it does not own. So the final row loses the attributes of the other systems. This is a classic MDM data-loss bug.

Source: https://paimon.apache.org/docs/master/concepts/primary-key-table/merge-engine/

partial-update keeps the latest NON-NULL value per column. So it ignores a later NULL for that column. To erase a value on a retract, you need sequence.field + partial-update.remove-record-on-delete. You must turn this behaviour on.

Source: https://paimon.apache.org/docs/master/concepts/primary-key-table/merge-engine/

A fixed bucket count gives a deterministic, reproducible layout. This is important for small tables and tests. With dynamic bucketing, the layout (and the merge timing) is not reproducible from one run to the next.

Source: https://paimon.apache.org/docs/master/concepts/primary-key-table/merge-engine/

You define a PK table with PRIMARY KEY(...) NOT ENFORCED. The merge engine merges rows that share that key. Partitioning and indexes are optional. format-version is an Iceberg concept.

Source: https://paimon.apache.org/docs/master/concepts/primary-key-table/merge-engine/

deduplicate = last whole record. partial-update = latest non-null value per column. aggregation = per-column aggregates. first-row keeps the FIRST record for a key and ignores later ones, so D is false.

Source: https://paimon.apache.org/docs/master/concepts/primary-key-table/merge-engine/

For immutable append-only facts, the LSM, PK merges and compaction give you no benefit. A plain columnar lake table is simpler and cheaper. The merge engines matter only when rows update in place.

Source: https://paimon.apache.org/docs/master/concepts/overview/