Series
Open Table Formats
Iceberg and Delta Lake turned directories of files into real tables, and both have changed enormously since v1. This series covers how to choose between them, then how each format's metadata evolved from v1 to today.
3 posts
Apache Iceberg vs Delta Lake: No Fluff, Just the Decision
Two years ago, Iceberg vs Delta Lake was a feature bake-off: who has row-level deletes, who has time travel, who has schema evolution. That comparison is dead. Both formats now have deletion vectors, row lineage, column mapping, and merge-on-read - they arrived at the same place from opposite directions, Iceberg through a version dial and Delta through table features. So the real decision has moved. It's not 'which format has feature X' anymore; it's which ecosystem you want to live in, which catalog you bet on, how much of your stack is Databricks, and how much the new interop layer lets you defer the choice entirely. This is that decision guide - what genuinely differs, what doesn't, and how to pick without regret.
Read more →How Apache Iceberg Grew Up: The Metadata Format from v1 to v4
Most engineers meet Iceberg as 'the format that gives you ACID on object storage' and stop there. But the format has a version dial - format-version 1, 2, 3, 4 - and each turn of it solved a real problem the last one left open. v1 made a table atomic on immutable files but could only append and overwrite whole files. v2 added row-level deletes, sequence numbers, and branches. v3 gave every row a stable identity, replaced position deletes with deletion vectors, and added richer types and encryption. v4 is the quiet refactor: relative paths, typed statistics, and the death of the file-system catalog. This is the continuity - what each version inherited, what it changed, and why.
Read more →How Delta Lake Grew Up: From the Transaction Log to Table Features
Most engineers meet Delta Lake as 'the thing that gives Spark ACID transactions' and stop there. But underneath is one idea - an append-only transaction log of atomic commits - and a version story that looks nothing like Iceberg's clean dial. Delta grew by a ladder of protocol versions (writer versions 1 to 7, reader versions 1 to 3), bolting one capability onto each rung, until the ladder itself became the bottleneck. Writer 7 and reader 3 replaced it with table features: an a-la-carte set of named capabilities a table opts into individually. This is how the log works, how the ladder was climbed, and how deletion vectors, row tracking, column mapping, clustering, variant, type widening, and catalog-managed tables all hang off that pivot.
Read more →