Why Data Lakes Broke, and Delta Was Born

Module: Beginners | Duration: ~20 min | Lesson: 1 of 7

Priya runs the nightly job that loads orders into TheWorldShop's data lake. It writes Parquet files to s3://theworldshop/orders/, and the morning dashboard reads them back. Simple.

One night the job dies halfway. Spark had written 60 of 200 files before the cluster lost a node. Nobody notices until 8am, when the finance dashboard shows revenue down 70 percent. The data isn't wrong. It's half there. There's no "the write didn't finish" flag anywhere, because on a plain Parquet lake, a table is just whatever files happen to be sitting in the folder right now.

Priya didn't make a mistake. The storage model she inherited has no idea what "finished" means. That's the gap Delta Lake was built to close.

2. Concept Explanation

The data lake promise, and the fine print

Around 2014 the pitch was irresistible. Stop paying for a proprietary warehouse. Dump raw Parquet and ORC onto cheap object storage. Point Spark or Hive at it and run SQL. Storage and compute, finally separated.

The storage model underneath was deceptively simple. A "table" was a directory of files, and a Hive Metastore that remembered which directories existed.

s3://theworldshop/orders/
  order_date=2024-01-15/part-00000.parquet
  order_date=2024-01-15/part-00001.parquet
  order_date=2024-01-16/part-00000.parquet

To read the table, an engine listed the directory and opened every file. To write, it dropped new files in. That convention worked, until production load found the cracks.

The four cracks

No atomicity. A write is many files. If the job dies after file 60 of 200, those 60 files are now "in the table." Readers see a half-written state. There's no commit, so there's no "all or nothing."

No isolation. Two jobs writing the same partition don't coordinate. The Hive answer is "overwrite the directory," so the second writer silently clobbers the first. Last writer wins, and the rows the first writer added are just gone.

Unsafe schema changes. Add a column in the Metastore, and old Parquet files still don't have it. Read old and new files together and the old rows come back null, no error, no warning. Drop a column and re-add it with the same name but a new type, and you get silent corruption.

Slow planning at scale. To answer "which files hold January data," Hive lists every file in the partition. With millions of small files, listing alone takes minutes before a single byte of data is read.

The root cause is one missing word

Every one of those cracks traces to the same thing. A Hive-style lake has no table-level metadata. The "table" is a folder you've agreed to treat as a table. There's no atomic unit of change, no version history, no record of which files are really part of the table at version N.

Delta Lake adds exactly that missing layer: an ordered, atomic, file-based transaction log that sits next to your Parquet and turns "a folder of files" into a real table.

Where Delta came from

Delta Lake was built at Databricks and open-sourced in 2019. The design goal was blunt: keep your data in plain Parquet on cheap object storage, and add a transaction log that gives you the guarantees a warehouse has. ACID transactions. Time travel. Schema enforcement. Row-level UPDATE, DELETE, and MERGE. All on files you can still read with any Parquet reader.

You don't replace your storage. You add a _delta_log/ directory beside the data, and that log becomes the source of truth for what the table is.

3. Worked Example

Here's the failure from the Hook, made concrete. The old way:

# Plain Parquet overwrite, the way TheWorldShop did it in 2017
df.write \
  .mode("overwrite") \
  .partitionBy("order_date") \
  .parquet("s3://theworldshop/orders/")

# What actually happens on S3:
# 1. Spark deletes the old partition files   <- readers see them vanish NOW
# 2. Spark writes the new files              <- takes 2-3 minutes
# 3. A reader during step 2 sees a partial or empty partition

For those few minutes, the partition is wrong on disk and every reader sees it. Now the Delta way for the same write:

# Delta write
df.write \
  .format("delta") \
  .mode("overwrite") \
  .partitionBy("order_date") \
  .save("s3://theworldshop/orders/")

# What actually happens:
# 1. Spark writes the new Parquet files     <- invisible to readers
# 2. Delta commits ONE log entry naming those files as the new version
# 3. Readers flip to the new version atomically. No half state is reachable.

The data files are still Parquet. The difference is step 2: a single atomic log commit that says "version 7 of this table is exactly these files." Until that commit lands, readers stay on version 6. There's no moment where half the write is visible.

4. Your Turn

Exercise: TheWorldShop runs an hourly job that reads from Kafka and writes Parquet batches to a plain S3 data lake.

1. List three distinct ways the lack of atomicity could corrupt what a reader sees.
2. For each, describe what an analyst querying the table would actually observe.
3. The team wants to add a discount_pct column. On a plain Hive lake, what happens when a query reads old files (no column) and new files (with the column) together?

5. Real-World Application

This is not a hypothetical. Plain-lake atomicity bugs are the reason whole platform teams exist. Before Delta, Databricks customers routinely hit silent data loss from concurrent writes, multi-minute query planning from directory listing, and schema-drift bugs nobody could reproduce.

The fix shipped as a transaction log, and it spread fast. Today companies run Delta at petabyte scale under Spark, and the table format is the default storage layer on Databricks and increasingly elsewhere.

For your career: every Delta feature you'll learn (the log, time travel, MERGE, schema enforcement, deletion vectors) is a direct answer to one of the cracks above. Learn the cracks and the features stop feeling like trivia. They feel inevitable.

6. Recap + Bridge

A plain data lake is files in a folder with no atomic unit, no isolation, no schema safety, and slow planning. Delta Lake keeps the Parquet and adds the one thing that was missing: a transaction log that makes a folder behave like a table.

Coming up next: Lesson 2 opens up that log. We'll look inside _delta_log/, read an actual JSON commit, and watch how a sequence of small append-only files becomes the single source of truth for what your table is.