Batch Ingestion Patterns
Connector-based batch ingestion: full vs incremental, cursors, idempotency, the Singer spec, and quality gates.
Move data on a schedule, correctly — the Airbyte/Singer/Meltano/Fivetran connector model, full vs incremental extraction, incremental cursors, idempotency on ingest, the Singer spec, schema drift, and data quality gates.
Course content
- 01When Batch Beats StreamingFree
- 02The EL(T) Connector Model: Airbyte, Singer, Meltano, Fivetran🔒
- 03Full vs Incremental Extraction🔒
- 04Incremental Cursors: High-Watermark, CDC, Change-Tracking🔒
- 05Idempotency on Ingest: Upserts, Merge, Dedup Keys🔒
- 06The Singer Spec: Taps, Targets, State🔒
- 07Scheduling & Orchestration Touchpoints🔒
- 08Schema Drift in Batch Pipelines🔒
- 09Data Quality Gates at Ingest🔒
- 10Capstone: Build an Incremental Ingestion Pipeline for TheWorldShop🔒
What to learn next
Read the first chapter free
Start reading now — no account required for the free chapters.