Build your own mini Kafka · Stage 4 of 8

Fetch by offset

Implement Fetch (api 1) and ListOffsets (api 2) so a real consumer can read your log.

Return the raw record batches covering offsets >= fetch_offset in the records field; use the offset index to seek and skip batches below the requested offset.
Report high_watermark, last_stable_offset, log_start_offset.
ListOffsets answers earliest (ts −2) and latest (ts −1).

Tests: a manually-assigned consumer reads everything from offset 0, reads from a mid-log offset, sees nothing past the end, and get_watermark_offsets returns (0, N).

Hints

Hint 1

A Fetch request (api 1, flexible from v12) asks for a fetch_offset per partition. Return the raw record batches whose records cover offsets >= fetch_offset, as the 'records' bytes field of the response.

Hint 2

Use the offset index to seek near the fetch_offset, then skip whole batches whose last offset (baseOffset + lastOffsetDelta) is below fetch_offset. Respect partition_max_bytes so you don't return the whole log at once.

Hint 3

The response per partition also needs high_watermark and (v4+) last_stable_offset and (v5+) log_start_offset. ListOffsets (api 2) must answer earliest (timestamp -2) = log start and latest (-1) = end so get_watermark_offsets works.

Run it locally

uv run de start KFK-BYO/04-fetch-by-offset
uv run de test