Build your own mini Kafka · Stage 8 of 8
Replication & the high watermark
In a replicated cluster the high watermark (HW) is the offset up to which records are committed on enough replicas to be visible to consumers. On your single node there are no followers, so once a record is durably written (flush + fsync) the HW advances to the log-end offset.
- Track a per-partition HW; advance it on durable append.
- Serve Fetch / ListOffsets(latest) from the HW and never return records at or beyond it.
- Restore the HW on startup so it (and all offsets) survive a restart from the same data dir.
Tests: HW equals the produced record count; a consumer reads everything up to the HW; HW and data survive a restart.
Hints
Hint 1
Track a high watermark (HW) per partition: the offset up to which records are committed and visible. On a single node, advance HW to the log-end offset once each record is durably written (flush + fsync).
Hint 2
Serve Fetch and ListOffsets(latest) using the HW, not the raw log-end offset, and never return records at or beyond the HW. A consumer's get_watermark_offsets high value is your HW.
Hint 3
On startup, restore the HW from what is durably on disk (equal to the recomputed log-end offset), so offsets and the HW survive a restart from the same data dir.
uv run de start KFK-BYO/08-replication-high-watermark
uv run de test