Build your own mini Kafka · Stage 2 of 8
A segmented log with an offset index
Kafka stores each partition as an append-only log split into segments, with a sparse offset index mapping offsets to byte positions so reads can seek.
Give every (topic, partition) its own directory under the data dir. Store
record batches in <baseOffset>.log files and index entries in
<baseOffset>.index (8 bytes each: relative-offset int32, byte-position int32).
Tests produce records (via Stage 3's Produce path — the reference implements
both) and then inspect your .log/.index files directly: a segment named by
base offset 0, a non-empty index whose first entry maps offset 0 to position 0,
and index positions that fall inside the log.
Hints
Hint 1
Give each (topic, partition) its own directory and store records in '<baseOffset>.log' files (20-digit zero-padded). Keep a parallel '<baseOffset>.index' of 8-byte entries: (relative_offset int32, byte_position int32).
Hint 2
Write a sparse index: add an entry for the first batch (offset 0 -> position 0) and then roughly every few KB. The byte position must be where that batch STARTS in the .log file.
Hint 3
Roll to a new segment file (named by the next base offset) once the active one passes a size threshold. On startup, scan existing segments to recompute the log-end offset.
uv run de start KFK-BYO/02-segmented-log-index
uv run de test