Build your own mini table format · Stage 1 of 7
Parquet files + a JSON manifest
A table format is just metadata that points at data files. In this stage you split the two: row data goes into immutable Parquet files; a tiny JSON manifest records which files make up the table.
You edit tableformat.py. The provided objectstore.py is a key→bytes store over
a local directory — treat it like S3. Do not edit objectstore.py.
Implement
Table.create(store, name, schema) -> Table— register a brand-new, empty table. Fail if it already exists. Write the first metadata object at key"<name>/metadata/v1.json".Table.load(store, name) -> Table— open an existing table (raise if missing).table.append(arrow_table) -> snapshot_id— write the rows to a new Parquet file under"<name>/data/<something>.parquet", then record that file in a new metadata object. Return the new snapshot id (an int).table.scan() -> pyarrow.Table— read all data files listed in the current metadata and return their concatenation as a singlepyarrow.Table.
Rules that make it a format, not a heap of files
- Data and metadata are separate objects. The manifest is JSON and lists data
files by key, with at least
pathandnum_rows. The tests open your JSON manifest and your Parquet files directly with a real reader. - Data files are immutable.
appendnever rewrites an existing file; it adds a new one. - Each metadata object is a new version
v<N>.json; the "current" table is the highest version present.
API you are starting (later stages extend the same file)
Table.create(store, name, schema) -> Table
Table.load(store, name) -> Table
table.append(arrow_table) -> snapshot_id
table.scan(filter=None, snapshot_id=None) -> pyarrow.Table
table.snapshots() -> list[Snapshot] # stage 2
table.delete(predicate) -> snapshot_id # stage 5
table.compact() -> snapshot_id # stage 7
table.expire(older_than_ms) -> int # stage 7
Run de test to check this stage. Later stages re-run these tests as a
regression suite, so keep append/scan working.
Hints
Hint 1
Metadata is just JSON you write with store.put_if_absent('<name>/metadata/v1.json', ...). Data is a Parquet file you write with store.put('<name>/data/<uuid>.parquet', pq-bytes).
Hint 2
append: write one new Parquet file (pyarrow.parquet.write_table into a BytesIO), then write a NEW metadata object whose manifest = old manifest + {path, num_rows}. The current version is the highest v<N>.json present.
Hint 3
scan: read the current metadata, pq.read_table(store.path(key)) for every manifest entry, and pyarrow.concat_tables(...) them. For an empty table return an empty table.
uv run de start LKH-BYO/01-parquet-manifest
uv run de test