Kafka Streams

A KStream models an unbounded sequence of independent facts. Each record stands alone. A KTable models the current value per key. It reads the topic as a changelog, so a new record for an existing key replaces the old value.

Source: https://kafka.apache.org/documentation/streams/concepts

The stream drives the join. Each incoming KStream record gets the current KTable value for its key. A KTable update alone produces no output. For this reason, orders (stream) enriched with customers (table) emit on each order.

Source: https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#kstream-ktable-join

KStream-KTable joins require co-partitioning. This means the same key and the same partition count, so matching keys land on the same partition and task. A GlobalKTable does not have this requirement, but it replicates the whole table to every instance.

Source: https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#joining

A compacted changelog topic backs each state store. After a restart, Kafka Streams restores the local store by replaying the changelog. The same happens on another instance after a rebalance. This makes stateful operations fault-tolerant.

Source: https://kafka.apache.org/documentation/streams/architecture

TopologyTestDriver runs the full topology in-process, without a broker or RocksDB. You send inputs and read outputs with controlled event time. Tests are fast and deterministic. They need no sleeps and no cluster.

Source: https://kafka.apache.org/documentation/streams/developer-guide/testing.html

Kafka Streams fully replicates a GlobalKTable to every instance. A KStream-GlobalKTable join can therefore look up by a key derived from the stream record, and it needs no co-partitioning. The tradeoff is the memory and replication cost. For this reason, it suits smaller reference tables.

Source: https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#kstream-globalktable-join

Event-time windowing puts each record in a window by its own timestamp. Results are correct even when records arrive out of order or late. With processing time, the results depend on nondeterministic arrival timing.

Source: https://kafka.apache.org/documentation/streams/core-concepts#streams_time

EOS v2 puts the input offset commit, the output produce, and the state-store changelog write in one transaction. When a failure occurs, the whole unit aborts and the application retries it. Each input then affects the output and state exactly once.

Source: https://kafka.apache.org/documentation/streams/core-concepts#streams_processing_guarantee