Kafka Connect & CDC

Debezium emits op='r' for every existing row during the initial snapshot, before streaming begins. Downstream logic should treat it like an initial insert. c/u/d are create/update/delete.

Source: https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-events

After a delete, Debezium emits a tombstone (key present, value null). Then a log-compacted topic can physically drop the key. The tombstone carries no new state. SCD2 derivation skips it, because the preceding op='d' already closed the row.

Source: https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-delete-events

A connector is a configured integration. It divides the work into tasks (for example per table or per partition range), and the workers run the tasks. tasks.max limits how many tasks run.

Source: https://kafka.apache.org/documentation/#connect_concepts

Logical decoding (which Debezium uses) requires wal_level=logical plus a replication slot. The slot retains WAL until the connector consumes it. So a stalled consumer can fill the disk. This is a key operational caveat.

Source: https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#setting-up-postgresql

SCD2 keeps history. An update closes the earlier version (valid_to = change ts, is_current=false). It opens a new current version from the after image. Overwriting loses history. If you leave the old row open, you get two current versions.

Source: https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-update-events

Services write domain events to an outbox table in the SAME transaction as the business change. Debezium streams the outbox. Consumers get a stable, curated event contract instead of raw internal tables. The write is atomic with the state change.

Source: https://debezium.io/documentation/reference/3.0/transformations/outbox-event-router.html

A DLQ routes records that fail (for example, at deserialization) to a separate topic. The connector keeps running. Then you inspect or replay the DLQ. You do not block the whole task on a poison record.

Source: https://kafka.apache.org/documentation/#connect_errorreporting

The envelope carries op, before, after and source. before is null for inserts/reads. after is null for deletes. source holds the LSN, the commit timestamp and the table identity. You use them to order and audit events.

Source: https://debezium.io/documentation/reference/3.0/connectors/postgresql.html#postgresql-events