Operating & troubleshooting a cluster

When a replica goes offline, the partition is under-replicated (ISR < replicas). When you restore the broker, the replica catches up and rejoins the ISR, and URP returns to 0. Or you can reassign the replica to a healthy broker.

Source: https://kafka.apache.org/documentation/#monitoring

acks=all requires the write to reach all in-sync replicas, and min.insync.replicas sets the floor. When the ISR is below that floor, the broker refuses the write (NOT_ENOUGH_REPLICAS). It does not risk a commit with too little durability.

Source: https://kafka.apache.org/documentation/#configuration

If the group is stuck at a bad position, reset its offsets to a sane point, so it can make progress. A blind reset to earliest can reprocess huge volumes. Pick the smallest point that unblocks it.

Source: https://kafka.apache.org/documentation/#basic_ops_consumer_group

retention.ms=1000 deletes data after one second. Any consumer that is not perfectly caught up loses records. The fix is to set retention.ms to a correct (much larger) value.

Source: https://kafka.apache.org/documentation/#configuration

The broker returns advertised.listeners to clients. If clients cannot reach that address, they get metadata but cannot connect. Set advertised.listeners to a host:port that clients can reach.

Source: https://kafka.apache.org/documentation/#configuration

A throttle that is too low limits the replication bandwidth that the move can use, so the move never catches up. Raise the throttle (or remove it). --verify also clears the throttle when the move completes.

Source: https://kafka.apache.org/documentation/#rep-throttle

A lower floor lets writes through with fewer in-sync replicas. This weakens durability. Prefer to restore replicas. Lower min.insync.replicas only as a conscious availability decision, which is usually temporary.

Source: https://kafka.apache.org/documentation/#configuration

A fix is only a fix if health is measurably restored. After the change, you verify the specific metric (URP=0, ISR >= min.insync, lag draining, retention >= floor, endpoint reachable, reassignment complete).

Source: https://kafka.apache.org/documentation/#monitoring