Event & topic design

Kafka guarantees order only within a partition. To keep all events for one entity in order, they must share a partition. For this reason, the key choice (key -> partition) is the ordering contract.

Source: https://kafka.apache.org/documentation/#intro_concepts_and_terms

For non-null keys, the default partitioner uses murmur2(key) % numPartitions. The same key always goes to the same partition (if the partition count stays fixed). The partitioner distributes a null key round-robin/sticky.

Source: https://kafka.apache.org/documentation/#producerconfigs_partitioner.class

company_id has low cardinality and is skewed. All events of the biggest tenant hash to one partition, which limits its throughput and lag. Key by the high-cardinality entity id. Then the load of a big tenant spreads across partitions, and each entity stays ordered.

Source: https://kafka.apache.org/documentation/#intro_concepts_and_terms

Log compaction retains the most recent value per key. A compacted topic keyed by customer id is therefore a changelog of current state. A KTable or consumer can rebuild it from the beginning. Time-retained event topics keep history. They do not give a clean view of the latest state.

Source: https://kafka.apache.org/documentation/#compaction

The mapping key -> partition is murmur2(key) % numPartitions. If numPartitions changes, the keys map to new partitions. New events of a key can land on a different partition than its old events. This breaks per-key ordering across the change. Size the partitions for future load up front.

Source: https://kafka.apache.org/documentation/#intro_concepts_and_terms

An event topic records independent facts and Kafka retains it by time (your fact feed). An entity topic is a compacted changelog keyed by entity id. It keeps the latest state per key (your dimension or KTable source).

Source: https://kafka.apache.org/documentation/#compaction

Each topic has partitions with a fixed overhead (files, replication, controller metadata). Thousands of tenant topics multiply this overhead and strain the cluster, even though isolation is strong. A shared, well-keyed topic usually scales better.

Source: https://kafka.apache.org/documentation/#intro_concepts_and_terms

Keying by the entity id puts all events of one entity on one partition (ordering). It also distributes the many entities of a big tenant across partitions (no hot spot). A random key per event destroys ordering. A null key round-robins. company_id creates a hot spot.

Source: https://kafka.apache.org/documentation/#producerconfigs_partitioner.class