Apache Kafka · KFK-06
Schema Registry & contracts
- 5 h
- requires KFK-05
From your world. dbt contracts and conformed dimensions, enforced on the wire
Evolve the order schema without breaking consumers
The platform publishes an order event. Version 1 of the Avro value schema is:
{"type": "record", "name": "Order", "namespace": "de.commerce",
"fields": [
{"name": "id", "type": "long"},
{"name": "amount", "type": "double"}
]}
Finance now wants a currency on every order. Hundreds of consumers already read version 1. They
will not all redeploy today. The subject uses the default BACKWARD compatibility mode.
Evolve the schema so that the registry accepts it and old consumers keep working.
Your task — implement solution.py
Implement three functions:
base_schema() -> dict— return the version-1 schema exactly as above.evolved_schema() -> dict— return a BACKWARD-compatible version-2 schema that adds acurrencystring field. (Think about what a new field needs, so that a reader can still read an old record.)produce_and_read(bootstrap_servers, topic, orders) -> dict— the input is a list of(id, amount, currency)tuples. Do these steps:- Serialize each order with your evolved schema and produce it to
topic. - Consume the records back. Decode each record twice. Decode it once with the evolved (new) reader schema and once with the base (old) reader schema.
- Return a dict:
Both lists must be ordered by{ "new_reader": [{"id": ..., "amount": ..., "currency": ...}, ...], "old_reader": [{"id": ..., "amount": ...}, ...], # no currency key }idascending.
- Serialize each order with your evolved schema and produce it to
Serialize with fastavro.schemaless_writer into bytes. Send those bytes as the Kafka message value.
Decode with fastavro.schemaless_reader. Pass the writer schema (evolved) and the chosen reader
schema.
Why this proves the point
The old_reader list decodes new-schema bytes with the old schema. A consumer that you have not
upgraded yet does exactly this in production. If your evolution is BACKWARD-safe, these decodes
succeed and ignore currency. Suppose you added a required field instead. Then the old reader
would fail. The registry would reject that change.
Constraints
- Do not drop or rename
idoramount. currencymust be a plainstringfield with a default (not a union/nullable).- No Docker. Tests run against a native Kafka broker and the pure-Python Avro compatibility checker.
Starter code
"""Starter — not implemented yet (must FAIL the tests: gate G1)."""
def base_schema():
# TODO: return the version-1 Order schema exactly as in the README.
return {}
def evolved_schema():
# TODO: return a BACKWARD-compatible version-2 schema adding `currency`.
return {}
def produce_and_read(bootstrap_servers, topic, orders):
# TODO: produce each order serialized with the evolved schema, then read
# every record back with BOTH the new and the old reader schema.
return {"new_reader": [], "old_reader": []}
Hints
Hint 1
Under BACKWARD, a new field must carry a default so old records can be read with the new schema. Add currency with default 'USD'; do NOT drop or rename id/amount.
Hint 2
Serialize each order with the NEW schema using fastavro.schemaless_writer, prepending nothing (plain schemaless binary is fine for this exercise).
Hint 3
Read every message back twice: once with the NEW reader schema (currency present) and once with the OLD reader schema (currency ignored). Both must succeed — that proves old consumers survive the evolution.
uv run de start KFK-06/01-evolve-order-schema
# edit workspace/KFK-06/01-evolve-order-schema/
uv run de test