CoursesChallenges

Backfill Strategies for Lakehouses

Partition-aware, idempotent backfills into Iceberg/Delta alongside live streams, with reconciliation.

Load history without breaking the present — idempotent writes into Iceberg/Delta, partition-aware backfills, running a backfill alongside a live stream, dual-pipeline reconciliation, watermark resets, lambda/kappa convergence, and verification.

Advanced10 chapters· 3h 30m· in Ingestion & Transport

Course content

  1. 01The Backfill Problem: History Without Breaking the PresentFree
  2. 02Idempotent Writes Into Iceberg/Delta🔒
  3. 03Partition-Aware Backfills🔒
  4. 04Running a Backfill Alongside a Live Stream🔒
  5. 05Dual-Pipeline Reconciliation🔒
  6. 06Watermark Resets & Reprocessing🔒
  7. 07Lambda vs Kappa, and Convergence🔒
  8. 08Backfilling Without Blowing the Budget🔒
  9. 09Verifying a Backfill: Counts, Checksums, Spot Checks🔒
  10. 10Capstone: Backfill Two Years of TheWorldShop Orders🔒

Prerequisites

↗Batch Ingestion Patterns

What to learn next

↗Designing a Production Ingestion Platform· next

Read the first chapter free

Start reading now — no account required for the free chapters.

Start: The Backfill Problem: History Without Breaking the Present →More in Ingestion & Transport