Producers — batching, idempotence, ordering under retries

The idempotent producer uses a producer id and per-partition sequence numbers. So the broker drops duplicates and out-of-order sequences. The setting also pins acks=all and limits in-flight requests.

Source: https://kafka.apache.org/documentation/#producerconfigs_enable.idempotence

Idempotence preserves ordering only up to 5 in-flight requests for each connection. Above 5, a retried request can arrive in a different order than later successful requests.

Source: https://kafka.apache.org/documentation/#producerconfigs_max.in.flight.requests.per.connection

Larger, compressed batches (linger.ms, batch.size, and compression) raise throughput. Idempotence and a key for each record keep the order per key. They also prevent duplicates on retry. All four settings apply.

Source: https://developer.confluent.io/courses/architecture/producer-hardware/

Idempotence is off and there is more than 1 in-flight request. In this case, a retry can arrive after a later successful request. Idempotence (and in-flight <= 5) prevents this with sequence numbers.

Source: https://kafka.apache.org/documentation/#semantics

Keyed records are hash-partitioned. So one key always maps to one partition. This is the basis of per-key ordering. Records without a key use sticky batching across partitions for throughput.

Source: https://kafka.apache.org/documentation/#producerconfigs

Idempotence requires the safe defaults and sets them. These are acks=all, retries that are effectively infinite but bounded by the delivery timeout, and in-flight <= 5. Compression and linger are independent throughput settings that you choose.

Source: https://kafka.apache.org/documentation/#producerconfigs_enable.idempotence