Hudi & the format landscape

Hudi maps each record key to a file group with an index (bloom / Record Level Index). So an upsert touches only the needed files. It does not scan or join the whole table. The timeline of Hudi powers incremental "what changed since T" queries. Hidden partitioning is Iceberg. Unity is Databricks/Delta.

Source: https://hudi.apache.org/docs/indexes/

Hidden partitioning and partition evolution of Iceberg let you change the layout as metadata (no data rewrite). The widest set of engines reads Iceberg. Delta leans toward Databricks. The upsert and index machinery of Hudi is overhead for an engine-neutral lake that is mostly analytic.

Source: https://iceberg.apache.org/docs/1.10.0/partitioning/

Delta is the native format on Databricks, with the best support: Unity Catalog governance, liquid clustering (CLUSTER BY), and Lakeflow pipelines. Databricks supports Iceberg and Hudi. But Delta is the first-class citizen there.

Source: https://docs.delta.io/latest/delta-clustering.html

UniForm (Delta) and XTable translate metadata between formats. Then you can read a single data copy as Iceberg, Delta or Hudi. Interop solves the need "another engine must read it". It does not need a costly migration or duplicated storage.

Source: https://xtable.apache.org/

The strengths of Hudi (indexes, upserts, incremental pulls) target mutable and CDC workloads. For append-only analytics, you pay for machinery that you do not need. Iceberg and Delta give a simpler path. Choose the format whose strength matches your write/read pattern.

Source: https://hudi.apache.org/docs/concepts/

Merge-on-Read writes change files or delete files and merges them at read time. So writes are cheap and fast, but reads do extra work (until compaction). Copy-on-Write rewrites data files on update. Writes cost more and reads cost less. The trade-off concept is the same across formats.

Source: https://hudi.apache.org/docs/concepts/

XTable (formerly OneTable) writes the metadata of one format from another. So the same underlying data files can appear as Iceberg, Delta or Hudi. It is interop. It is not a competing format.

Source: https://xtable.apache.org/