Apache Kafka · KFK-08
Kafka Streams
- 7 h
- requires KFK-07
From your world. dbt joins between a fact and a dimension, but continuous
Explain this to a teammate who knows dbt well. Cover how a Kafka Streams KStream-KTable join enriches orders with customers. Also cover why it is not just "a join in SQL."
Rubric — a good answer covers:
KStream= append-only facts (orders).KTable= latest value per key, a changelog (customers, a conformed dimension).- The stream drives the join. Each order looks up the current customer. A change in the table alone emits nothing.
- Co-partitioning (same key, same partition count) is required. The other
option is a
GlobalKTable. - State lives in a state store that a compacted changelog topic backs. A crashed instance recovers by replaying the changelog.
- Event-time windows (not processing time) keep aggregations correct when records arrive in a different order.
- TopologyTestDriver runs the topology in-process (no broker, no RocksDB), deterministically in event time. This is how you unit-test the logic.
- Analogy: it is the fact-to-dimension lookup in dbt, but continuous and incremental. It always uses the latest dimension value at the time the event arrives.