Cluster configuration & partition reassignment

The first broker in the replica list is the preferred leader. The other brokers are followers. They replicate from the leader and stay in the ISR. After a disruption, preferred-leader election tries to make that first broker the leader again.

Source: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

Replication exists so that the data survives the loss of a broker. Suppose two replicas of one partition are on the same broker. Then a single failure takes out two copies, and this defeats the purpose. Assignments must use distinct brokers.

Source: https://kafka.apache.org/documentation/#replication

If you keep an existing replica (broker 0), the data of that copy stays in place. Kafka copies only the 1 -> 2 replica over the network. Minimal-movement reassignment keeps as many current replicas as the constraints allow. The order [2, 0] also keeps broker 0, so it moves the same data as [0, 2]. The order changes only the preferred leader. It does not cause a full copy.

Source: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

Rack awareness spreads replicas across racks. With rf <= number of racks, each replica should be on a distinct rack. Then the failure of a whole rack (or AZ) cannot take all replicas of a partition offline.

Source: https://kafka.apache.org/documentation/#basic_ops_racks

A reassignment copies partition data between brokers. A throttle limits this traffic, so it does not saturate the network and degrade production. But if the throttle is too low, the reassignment may never complete (a KFK-14 incident).

Source: https://kafka.apache.org/documentation/#rep-throttle

--verify compares the in-progress reassignment with the target JSON. It reports when the move of each partition is complete (and removes any throttle). You run it repeatedly until it is done.

Source: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

acks=all means the leader waits for all in-sync replicas. min.insync.replicas=2 requires at least 2 replicas in the ISR before the broker accepts the write. If the ISR drops below 2, the producer gets NOT_ENOUGH_REPLICAS. This avoids a write with too little durability.

Source: https://kafka.apache.org/documentation/#brokerconfigs

An assignment from scratch ignores the current placement. Most replicas move, and the cluster copies huge volumes of data at once. Prefer a minimal-movement plan (keep existing replicas) and throttle the move.

Source: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion