Build your own mini Kafka · Stage 6 of 8

Retention and log compaction

Topics carry a cleanup.policy (set through CreateTopics configs).

compact: keep only the latest record per key, preserving each survivor's original offset, WITHOUT changing the log-end offset.
delete (default): keep every version (optionally drop whole old segments by retention.bytes).

Compaction rewrites the log in place. Decode the records out of each v2 batch to find the key of each record.

Tests: a compacted topic returns one record per key (the newest value); the log end offset is unchanged; a delete-policy topic keeps all versions.

Hints

Hint 1

Read the topic's cleanup.policy (set via CreateTopics configs). For 'compact', keep only the LATEST record per key; for 'delete', keep everything (optionally drop old segments by retention.bytes).

Hint 2

Compaction rewrites the log: scan all batches, remember the latest (value, offset) per key, then rebuild one record per surviving key IN OFFSET ORDER, preserving each record's original absolute offset as its batch baseOffset.

Hint 3

Compaction must NOT change the log-end offset (new records still get the next offset). Decode records from a v2 batch by walking the varint-length-prefixed records after the 61-byte header.

Run it locally

uv run de start KFK-BYO/06-retention-compaction
uv run de test