DE Mastery GitHub

Apache Kafka · KFK-04

Consumers & groups — manual commits, at-least-once, no loss across crashes

From your world. SOX completeness: a crash must never drop an unprocessed event

Prompt A. "Explain the difference between at-most-once and at-least-once only in terms of WHEN you commit the offset, compared with when you do the work. Then explain why at-least-once plus an idempotent downstream write gives you effectively-correct results after a crash."

Prompt B. "A consumer with auto-commit crashes in the middle of a batch. Your teammate is surprised that some records 'disappeared'. Explain exactly what happened. Give the one-line config change and the code change that would have prevented the loss."

Rubric — a strong explanation hits these (5–7):

  • Defines the committed offset as the group's per-partition "handled up to here".
  • Commit-before-work = at-most-once (loss on crash). Commit-after-work = at-least-once (dup on crash).
  • Explains auto-commit advancing the offset independent of processing → the loss trap.
  • States that at-least-once never loses data and reprocesses from the last commit.
  • Explains an idempotent sink (upsert by business key) absorbing the duplicate.
  • Describes groups dividing partitions (one per consumer) and rebalancing for scale/failover.
  • (Bonus) Mentions KIP-848 incremental rebalance and that exactly-once needs transactions.