Transactions & exactly-once — read-process-write, read_committed, fencing

When you commit the input offsets inside the transaction, consume+produce is atomic. After a crash, Kafka either rolls back both or commits both. So Kafka never processes an input twice and never drops one.

Source: https://kafka.apache.org/documentation/#semantics

read_committed makes the consumer skip records that belong to aborted or not-yet-committed transactions. read_uncommitted (the default) shows them, which breaks exactly-once.

Source: https://kafka.apache.org/documentation/#consumerconfigs_isolation.level

Fencing uses transactional.id + epoch. A stable id lets a new instance fence the old one. With random ids, the two instances have different ids and both could commit. This causes duplicates.

Source: https://kafka.apache.org/documentation/#producerconfigs

Aborting discards the transaction atomically. read_committed consumers never see its records. This is the atomicity half of exactly-once.

Source: https://kafka.apache.org/documentation/#semantics

You need all four. Idempotence is the base. A stable transactional.id gives fencing. Offsets committed in the transaction make consume+produce atomic. read_committed hides aborted and in-flight records.

Source: https://developer.confluent.io/courses/architecture/exactly-once/

Suppose a downstream idempotent upsert removes duplicates for you. Then at-least-once avoids the overhead and latency of the transaction coordinator. Transactions are best when you need atomicity across Kafka topics.

Source: https://developer.confluent.io/courses/architecture/exactly-once/