CDC at Scale: Patterns & Pitfalls

Outbox pattern, ordering, schema evolution, DLQ/replay, exactly-once sinks, and GDPR erasure for CDC.

The hard half of CDC — the outbox pattern, dual-write hazards, schema evolution propagation, ordering and per-key guarantees, dead letter queues and replay, exactly-once into lakehouse sinks, and GDPR erasure propagation across a streaming pipeline.

Advanced11 chapters· 3h 50m· in Ingestion & Transport

Course content

  1. 01The Dual-Write Hazard, Revisited at ScaleFree
  2. 02The Outbox Pattern🔒
  3. 03Ordering & Per-Key Guarantees Across Topics🔒
  4. 04Schema Evolution Propagation Through the Pipeline🔒
  5. 05Dead Letter Queues & Replay🔒
  6. 06Exactly-Once Into Sinks (Iceberg/Delta/Warehouse)🔒
  7. 07Initial Loads & Incremental Snapshots at Scale (Signaling)🔒
  8. 08GDPR Erasure Propagation Through CDC🔒
  9. 09Monitoring CDC: Lag, Slot Growth, Connector Health🔒
  10. 10Failure Modes & Recovery (slot bloat, snapshot restarts)🔒
  11. 11Capstone: Design a Resilient CDC Pipeline for TheWorldShop🔒

Prerequisites

What to learn next

Read the first chapter free

Start reading now — no account required for the free chapters.