How Delta Lake Grew Up: From the Transaction Log to Table Features

Most engineers meet Delta Lake as a one-line answer: "it's the thing that gives Spark ACID transactions and time travel on object storage." That's true, and it's also where most people stop. But Delta has a whole model underneath - an append-only transaction log - and a version story baked into every table that looks nothing like the clean dial you may have seen in Apache Iceberg.

Where Iceberg evolved through a single format-version integer from 1 to 4, Delta grew by a ladder of protocol versions - writer versions 1 through 7 and reader versions 1 through 3 - bolting one capability onto each rung. Then, at the top of that ladder, it did something more interesting than add another rung: it replaced the ladder with table features, an a-la-carte model where a table opts into named capabilities one at a time.

To follow that arc you need the one mental model the rest of this post hangs on: the log. So we start there, walk the version ladder, hit the pivot to table features, and then tour every capability that pivot unlocked. By the end you'll know what's actually on disk and why.

The primer: a Delta table is a log, not a folder

The instinct everyone brings from Hive is that a table is a directory: point at a path, list the Parquet files under it, that's your table. Delta breaks that instinct, but in a different way than Iceberg does. An Iceberg table is a tree of immutable metadata with one movable catalog pointer. A Delta table is an ordered, append-only log of atomic commits sitting next to the data files.

The data files are ordinary Parquet, stored in the table's root directory (or partition subdirectories). But which files are live, what the schema is, what the table can do - none of that comes from listing the directory. It all comes from the log in _delta_log/. The log is the source of truth; the directory is just a bag of files.

Here is a real Delta table on disk:

/mytable/_delta_log/00000000000000000042.json               <- commit 42 (JSON, one action per line)
/mytable/_delta_log/00000000000000000042.checkpoint.parquet <- checkpoint: reconciled state at v42
/mytable/_delta_log/00000000000000000043.json               <- commit 43
/mytable/_delta_log/00000000000000000044.json               <- commit 44
/mytable/_delta_log/00000000000000000045.json               <- commit 45
/mytable/_delta_log/_last_checkpoint                         <- pointer to the newest checkpoint
/mytable/part-00000-....snappy.parquet                       <- data (ordinary Parquet)
/mytable/deletion_vector-0c6cbaaf-....bin                    <- deletion vector (soft-deleted rows)
/mytable/_change_data/cdc-00000-....snappy.parquet           <- change data feed files

Two rules make this an ACID table, and they're the whole pitch:

1. A commit is the atomic creation of one log file. To move the table from version v-1 to v, a writer writes ..0000v.json - and only one writer can win that filename. The log is append-only; writers must never overwrite an existing entry. On a filesystem or object store that offers a put-if-absent primitive, that single guaranteed-unique file creation is the entire contended operation. Everything else - writing the new Parquet files - happens optimistically and in parallel, before the commit, with no locking.
2. Readers reconstruct state by replaying the log. A snapshot at version v is defined by applying every commit from 0 to v in order. Delta uses multi-version concurrency control (MVCC): rather than mutate files in place, it keeps old files around and each commit records which files to logically add and remove. A reader picks one consistent version and only reads the files live at that version.

That's the core difference worth holding in your head. Iceberg swaps a pointer to a new root; Delta appends a new entry to a log. Both get you atomic commits on immutable files - Delta just gets there through an ordered sequence of transactions instead of a tree.

What lives in a commit: actions

Each .json commit is newline-delimited JSON, one action per line. An action changes one aspect of table state. The full vocabulary is small:

The add action is the heart of it. Every live file in the table is an add that hasn't been cancelled by a later remove:

{
  "add": {
    "path": "date=2017-12-10/part-000...c000.snappy.parquet",
    "partitionValues": {"date": "2017-12-10"},
    "size": 841454,
    "modificationTime": 1512909768000,
    "dataChange": true,
    "stats": "{\"numRecords\":1,\"minValues\":{...},\"maxValues\":{...}}"
  }
}

The dataChange flag is a quiet but important detail: set it to false and the commit is declared to only rearrange existing data (a compaction) or add statistics, not change logical contents. A streaming reader tailing the log skips those commits entirely - they can't produce new rows.

Checkpoints: so readers don't replay 40,000 commits

Replaying the entire log from commit 0 would get expensive fast. So writers periodically write a checkpoint: a Parquet file containing the reconciled state of the table - every live add, the current metaData and protocol, live txn and domainMetadata actions - with all the cancelled-out actions already removed. A reader that finds a checkpoint at version N can start there and replay only the handful of JSON commits after it.

Finding the newest checkpoint without listing a huge directory is the job of the _last_checkpoint file, which points near the end of the log. Because log files are zero-padded to 20 digits, a reader can then do a single lexicographically sorted listing to find every commit newer than that checkpoint.

Delta has three checkpoint shapes, and their evolution is itself part of the story:

• Classic - one file N.checkpoint.parquet.
• Multi-part - N.checkpoint.o.p.parquet split across p files. Now deprecated: they can't be written atomically, so a reader that catches a half-written set has to ignore it, and racing writers cause trouble.
• V2 (UUID-named) - N.checkpoint.<uuid>.json|parquet, which carries a checkpointMetadata action and references any number of sidecar Parquet files in _delta_log/_sidecars/ that hold the actual file actions. This is the modern form, and it's gated behind a table feature we'll get to.

Action reconciliation: the rules of replay

"Replay the log" needs precise rules, because commits contradict each other by design (an add here, a remove there). Reconciliation is short:

• The latest protocol wins. The latest metaData wins.
• For txn, the latest version per appId wins.
• For domainMetadata, the latest per domain wins; removed: true acts as a tombstone.
• A logical file is keyed by (path, deletionVector.uniqueId). Keep the newest reference to each. Surviving adds are the live files; surviving removes are tombstones that exist only so VACUUM knows what it may physically delete after the retention period (7 days by default).

Hold this picture - log of commits, periodic checkpoints, replay by reconciliation. Everything below is either a new action field or a new rule layered onto this same log. And the thing that tells a reader which rules apply is the protocol action.

The version ladder: how Delta bolted on capabilities

The protocol action is Delta's version dial. In its original form it carried just two integers:

{ "protocol": { "minReaderVersion": 1, "minWriterVersion": 2 } }

minReaderVersion is the lowest reader protocol a client must implement to read the table; minWriterVersion the lowest to write it. A client that supports less refuses the table rather than silently misreading it - the version is a contract, not a hint. Crucially, reader and writer versions advance independently, because many capabilities affect only how you write a table, not how you read it back.

For years, that's how Delta grew: each new capability raised a version number, and every rung inherited everything below it. Here is the actual ladder.

Writer versions stack up like this:

Reader versions are shorter, because most features are writer-only:

Look closely at that column-mapping row and you can see the whole problem with a linear ladder. Column mapping needs cooperation from both reader (version 2) and writer (version 5). So the reader and writer numbers don't line up, and a table that wants column mapping drags in everything on both ladders below those rungs - whether it uses those features or not. Want identity columns (writer 6) but not CDF or generated columns? Too bad; writer 6 implies you implement writer 4. The ladder is coarse: you buy capabilities in a bundle, and every new feature means a new global rung that every client must climb in lockstep.

By writer version 6, Delta had run out of room. The next feature couldn't just be "writer version 8," because that would force every writer in the world to implement a monotonic pile of unrelated capabilities to stay current. So the pivot wasn't another rung. It was a new mechanism.

The pivot: table features

Writer version 7 and reader version 3 change what the protocol means. Instead of a number implying a fixed bundle, the protocol action grows two lists of named capabilities:

{
  "protocol": {
    "minReaderVersion": 3,
    "minWriterVersion": 7,
    "readerFeatures": ["columnMapping", "deletionVectors"],
    "writerFeatures": ["columnMapping", "deletionVectors", "identityColumns", "rowTracking"]
  }
}

Now a table declares exactly which capabilities it uses. readerFeatures exists only at reader version 3; writerFeatures only at writer version 7. A client reads the table only if it implements every feature in readerFeatures; it writes only if it implements every feature in writerFeatures (and, because writing means reading first, all reader features too). Everything else it can ignore.

A few rules give this structure its shape:

• Two kinds of features. A writer-only feature (like appendOnly or identityColumns) affects only writes, so it appears only in writerFeatures. A reader-writer feature (like columnMapping or deletionVectors) changes the read path too, so it must appear in both lists. It is illegal to list a feature only in readerFeatures.
• Supported vs active. A feature being supported (its name is in the list) does not mean it is active. appendOnly can be supported while the table property delta.appendOnly is unset - meaning writers know to check that property before writing, but the table isn't actually append-only. A feature is active only when it's supported and its metadata requirements are met. This split is what lets a client know a feature might be in play without forcing it on.
• Legacy features fold in. Everything from the old ladder becomes a named feature: appendOnly, invariants, checkConstraints, generatedColumns, columnMapping, identityColumns, and so on. A table on writer version 2 with column invariants is equivalent to a writer-version-7 table with invariants in its writerFeatures. Upgrading is a metadata-only protocol action - no data rewrite - though the writer must be conservative and assume a feature was used unless it can prove otherwise across the table's whole history.

That last point is the elegance of the design. The version integers didn't die; they became a floor. (readerVersion 3, writerVersion 7) is the entry ticket to the feature system, and from there capability is a set, not a number. New features ship as new names, and old clients keep working on old tables because the feature list tells them precisely what they'd need to understand.

Here is the current feature vocabulary, grouped by whether it touches the read path:

The rest of this post walks the ones that matter, roughly in order of how much they change what's on disk.

Deletion vectors: merge-on-read, done as a bitmap

The single biggest capability the feature system unlocked is deletion vectors (deletionVectors, a reader-writer feature). Before them, deleting one row from a Parquet file meant copy-on-write: rewrite the entire file without that row and swap it in. On a wide file that's gigabytes rewritten to drop a handful of rows.

A deletion vector flips this to merge-on-read. Instead of rewriting the data file, a writer records a compressed bitmap of the row positions that are now logically deleted, and attaches it to the file's add action:

{
  "add": {
    "path": "part-00000-....parquet",
    "deletionVector": {
      "storageType": "u",
      "pathOrInlineDv": "ab^-aqEH.-t@S}K{vb[*k^",
      "offset": 4,
      "sizeInBytes": 40,
      "cardinality": 6
    },
    "stats": "{\"numRecords\": 1000, ...}"
  }
}

The reader loads the file, then skips the 6 rows the bitmap marks (cardinality is how many rows it removes). The bitmap itself is a RoaringBitmap - the same compressed 64-bit integer set Iceberg uses for its deletion vectors - and storageType says where to find it:

• 'u' - on disk, in a deletion_vector_<uuid>.bin file relative to the table. One .bin can hold DVs for files from many partitions, so DV files live at the table root, not in the partition tree.
• 'p' - on disk at an absolute path.
• 'i' - inline in the log itself, Z85-encoded, for tiny vectors.

There's one hard requirement worth calling out: when a file has a DV, its add action's stats.numRecords must be the physical record count of the Parquet file - not the count after deletes. The DV is what accounts for the difference. The logical file is now the pair (path, deletion vector id), which is exactly why reconciliation keys files on that tuple: the same Parquet file with two different DVs is two different logical files across versions.

If you read the Iceberg post, this is the same idea Iceberg reached in its format-version 3 - one bitmap per data file instead of a pile of tiny delete files. Two independent formats converged on the same answer.

Row tracking: a stable identity per row

Row tracking (rowTracking, writer-only) gives every row a durable identity that survives compaction - so you can build change feeds, do incremental processing, and reason about lineage. Delta's design is a clever two-tier scheme that mostly avoids storing an id per row.

Every row has two of each identifier:

• A fresh id/version, which uniquely identifies a row within one version of the table and may change on every write (even for untouched rows copied during an update).
• A stable id/version, which follows the same logical row across versions and rewrites.

And each is stored in one of two ways:

The default-generated id is computed, not stored - baseRowId + row_position - which is the same trick Iceberg uses for its _row_id. It's free but gets reassigned whenever OPTIMIZE moves a row to a new file. The materialized column is the escape hatch: when a writer needs a row's stable id to outlive a rewrite, it writes that id into the hidden column so the identity travels with the row. Read a row and the reader takes the materialized value if present, otherwise falls back to baseRowId + position. Row commit versions work identically, tracking the last version in which each row was inserted or updated.

Column mapping: rename and drop without rewriting

Column mapping (columnMapping, reader-writer) decouples a column's display name from its physical name on disk. Every column gets a stable physical name (col-<uuid>) and a 32-bit id, both stored in the schema metadata:

{
  "name": "revenue",
  "type": "long",
  "metadata": {
    "delta.columnMapping.id": 4,
    "delta.columnMapping.physicalName": "col-5f422f40-de70-45b2-88ab-1d5c90e94db1"
  }
}

In name mode the reader resolves columns by physical name; in id mode by the Parquet field_id. Either way, renaming a column is a metadata change to its display name - the physical name on disk never moves - and dropping a column just stops referencing its physical name. No data rewrite for either. It also frees tables from Parquet's column-naming restrictions (spaces, special characters), and it's a prerequisite for the Iceberg-compatibility features, since Iceberg is an id-based format.

The rest of the feature surface

The feature model turned "add a capability" from a protocol-wide event into a routine pull request. The result is a long tail of features, each self-contained. The ones you'll actually meet:

• Change Data Feed (changeDataFeed, writer-only). When delta.enableChangeDataFeed is set, data-changing operations also write cdc files under _change_data/, each carrying a _change_type column (insert, update_preimage, update_postimage, delete). A change reader consumes those files directly instead of diffing snapshots - and if a version has no cdc action, it treats that version's add/remove files as inserts and deletes.
• Clustering (clustering, writer-only). Liquid clustering: instead of rigid Hive-style partition directories, the table lists clustering columns in a domainMetadata action (domain: "delta.clustering"), and a background process incrementally co-locates rows with similar values. Files record which implementation clustered them via clusteringProvider. You can add or change clustering columns over time - impossible with physical partitioning - as long as the table isn't also partitioned.
• Type widening (typeWidening, reader-writer). Widen a column's type (int to long, float to double, decimal precision increases, date to timestamp_ntz) as a metadata change, with no file rewrite. Old files keep their narrow type; the reader up-casts on read, guided by a delta.typeChanges history recorded in the column metadata.
• TimestampNtz (timestampNtz, reader-writer). A timestamp without timezone (1970-01-01 00:00:00), which needs a reader that won't silently apply a zone.
• Variant (variantType, and variantShredding, reader-writer). A variant type for semi-structured data, stored in Parquet as a value + metadata binary pair. Shredding pulls hot paths out into typed Parquet columns so they can be data-skipped, without duplicating the bytes.
• Default columns (allowColumnDefaults), identity columns (identityColumns), generated columns (generatedColumns), CHECK constraints (checkConstraints), invariants (invariants), append-only (appendOnly) - the write-integrity family carried over from the old ladder, now each an independent opt-in.
• In-commit timestamps (inCommitTimestamp, writer-only). Instead of trusting the log file's filesystem modification time for time travel - which can drift or be rewritten - the writer records a monotonic inCommitTimestamp inside each commit's commitInfo, making "as of timestamp X" reliable and portable.
• V2 checkpoints (v2Checkpoint, reader-writer). Unlocks the UUID-named checkpoint-plus-sidecars form, and forbids the fragile multi-part checkpoints.
• VACUUM protocol check (vacuumProtocolCheck, reader-writer). A safety interlock: it forces VACUUM to run the writer protocol check, so an old client can't garbage-collect files that a newer feature still depends on.

There's also a quiet integrity feature worth knowing: version checksum files (N.crc), which a writer can emit alongside each commit recording the table size, file count, and full reconciled state at that version. A reader can independently recompute those numbers and compare - a cheap way to catch a log that's been tampered with or corrupted, since the log is supposed to be append-only.

Iceberg compatibility: one set of Parquet, two formats

Two features - icebergCompatV1 and icebergCompatV2 (both writer-only) - deserve their own note because they're the foundation of Delta UniForm ("Universal Format"). They don't convert anything themselves; they constrain how Delta writes so that the same Parquet files can also be described by an Iceberg metadata tree written alongside.

The constraints are what make the two formats agree on bytes. When icebergCompatV2 is active, a writer must:

• Enable column mapping in name or id mode (Iceberg is id-based).
• Assign Parquet field ids to nested element/key/value fields too.
• Materialize partition values into the Parquet files (Iceberg doesn't infer them from directory paths).
• Populate numRecords on every add.
• Not use deletion vectors, and restrict the schema to types both formats share.

The payoff: one physical copy of the data, readable as either a Delta table or an Iceberg table, so an Iceberg-only engine can query it without a conversion job. It's Delta's answer to the same interoperability pressure that shaped Iceberg's own evolution.

Catalog-managed tables: Delta reaches for a catalog

The newest frontier, catalog-managed tables (catalogManaged, reader-writer), is where Delta's log model starts to converge with Iceberg's catalog model - and it's worth understanding as the direction the format is heading.

Recall that a plain Delta commit relies on put-if-absent to make writing N.json atomic. That works when the storage layer offers it, but it's a weak foundation on systems that don't, and it makes the filesystem the arbiter of who won a commit. Catalog-managed tables move that authority to the catalog: the catalog, not the filesystem, decides whether a commit at version v succeeded.

The mechanics introduce a small vocabulary of commit states:

• A staged commit is written to _delta_log/_staged_commits/<v>.<uuid>.json - same content as a normal commit, but its mere existence proves nothing.
• A writer proposes a staged (or inline) commit to the catalog.
• The catalog ratifies exactly one proposal per version - that's the atomic decision point, now a catalog operation instead of a file creation.
• Ratified commits are later published by copying them to the normal _delta_log/<v>.json.

The consequence is strict: filesystem-based access to a catalog-managed table is not supported. Recent commits may live only in the catalog or in _staged_commits, so a client that just lists _delta_log/ would read a stale table. You must go through the catalog. This is the same lesson Iceberg's format version 4 encoded when it removed the file-system catalog entirely: on real object storage, a table needs a real transactional coordinator, not clever file naming.

How a read actually works

Putting it together, here's what a client does to read a Delta table at its latest version:

1. Read _last_checkpoint to jump near the end of the log.
2. Load the checkpoint it points at (V2 checkpoints may fan out into sidecar files) to get the reconciled state at that version cheaply.
3. List and replay the JSON commits newer than the checkpoint, applying action reconciliation: newest protocol and metaData win; keep the newest reference per (path, deletionVector.uniqueId).
4. Check the protocol. If readerFeatures lists anything the client doesn't implement, stop - the table uses rules the client can't honor.
5. Plan the scan. For each live add, use its per-column stats (minValues/maxValues/nullCount) to skip files the query predicate rules out. Resolve columns by physical name/id if column mapping is on.
6. Read data, apply deletion vectors. For each surviving file, read the Parquet and drop the row positions its deletion vector marks. Never emit an invalidated row.

Checkpoint, then commits, then stats-based file skipping, then per-file DV filtering. That funnel is why a log-shaped table can still answer a query over a petabyte without reading most of it.

Reading across versions

Like Iceberg, Delta is built so older tables keep working and the protocol is an explicit contract:

• A version-2 writer's table reads as a table-features table unchanged - the legacy version implies a fixed feature set, and upgrading to writer 7 / reader 3 is a metadata-only protocol action that names those same features.
• A reader refuses any feature it doesn't recognize in readerFeatures, rather than guessing. The feature list is exhaustive by design: unrecognized fields can be safely ignored (breaking changes must raise a version or add a feature), but unrecognized features mean "you can't correctly read this."
• The version integers are a floor, not a ceiling. (3, 7) just means "this table speaks the feature language"; what it actually requires is the union of its two feature lists.

What Databricks builds on top

Delta Lake the format is open source and vendor-neutral - everything above is in the public PROTOCOL.md. Databricks, which originated Delta, layers commercial capabilities on top of it, and it's worth knowing the line between the two:

• Unity Catalog is the productized catalog that sits behind catalog-managed tables - governance, lineage, and the commit-coordination authority the catalogManaged feature was designed for.
• Predictive optimization automates the maintenance the format enables - running OPTIMIZE, liquid clustering, and VACUUM on a schedule without a user wiring up jobs.
• Photon is a native vectorized query engine that reads Delta faster, but changes nothing about the on-disk format.
• UniForm is the packaged experience over icebergCompatV2 (and Hudi) - point external Iceberg engines at a Delta table with no conversion step.

The pattern is consistent: the format defines the mechanism (a feature, a metadata domain, a compatibility constraint), and the vendor sells the automation and governance around it. Nothing above requires Databricks to read or write a correct Delta table.

The throughline

Read Delta's history back to front and the arc is clean. It started with one idea - an append-only log of atomic commits, where a snapshot is just a replay - and that idea never changed. What changed was how the format grew. For six writer versions it grew by a ladder, each rung a bundled capability every client had to climb in lockstep, until the ladder's coarseness became the bottleneck. Then writer 7 and reader 3 replaced the ladder with table features: capability as an a-la-carte set instead of a monotonic number.

That pivot is the whole story. It's what let deletion vectors, row tracking, clustering, variant, type widening, and catalog-managed commits each ship as an independent opt-in, without forcing the entire ecosystem to move together - and it's why a format that began as "ACID for Spark" can now back merge-on-read deletes, row-level lineage, and catalog-coordinated commits across petabytes, all on the same one primitive it started with: append one file to the log, and let exactly one writer win.

If you want to build this understanding from the ground up - why data lakes broke, how transaction logs and snapshots really work, and where "ACID on a data lake" quietly stops holding - that's the open table formats curriculum, which puts Delta Lake side by side with Apache Iceberg. The two formats solve the same problem from opposite ends - a log versus a tree - and reading them together is the fastest way to understand either.