Design a Lakehouse

Every problem so far ended the same way: the data lands "in an Iceberg table" and the diagram moves on. The CDC pipeline merged change events into one. The batch warehouse built a star schema on one. The ad aggregator archived raw clicks to one. This chapter is the design of that table itself - the foundation every other pipeline writes into, and the reason they could wave their hands and trust it.

PhoenixWorld's data lives in two worlds that don't talk. A warehouse (think Snowflake) that BI trusts - transactional, consistent, fast - but expensive and a walled garden the ML team can't cheaply read from. And a data lake (Parquet files on S3) the ML team loves - cheap, open, readable by any engine - but with no transactions, no schema safety, no time travel, and the recurring nightmare where a job half-writes a partition and every dashboard reads garbage until someone notices. Two copies of the data, two bills, and two versions of "the truth" that quietly disagree. Mei's mandate is to collapse them into one storage layer that is somehow both. That is the lakehouse.

The trick at the center of it is deceptively unglamorous: ACID guarantees on top of object storage that provides none. S3 has no transactions, no locks, no multi-object atomicity, and historically was even eventually consistent. Yet the lakehouse promises atomic commits, concurrent writers, snapshot isolation, and time travel. It pulls this off entirely with metadata - a tree of small pointer files and a single atomic swap. Understand that tree, and where it strains under real workloads, and you understand the whole problem.

Step 1 - Understand the Problem and Establish Design Scope

"Build a lakehouse" can mean a table format, a catalog, or a whole platform. The clarifying questions scope it to the table layer and its maintenance.

Candidate: What are we replacing, and who reads the result?

Interviewer: A warehouse plus a separate data lake, today held as two copies. We want one open table layer on object storage that both BI (via Trino) and ML (via Spark) read directly, no copies.

Candidate: What guarantees does it have to provide that raw Parquet on S3 doesn't?

Interviewer: ACID commits, so a reader never sees a half-written table. Snapshot isolation. Schema evolution without rewriting the world. Time travel and rollback. And row-level updates and deletes.

Candidate: Who writes to it, and concurrently?

Interviewer: Several writers at once: a streaming job upserting CDC changes every minute, a nightly batch building marts, and occasional backfills. They can't corrupt each other.

Candidate: What scale?

Interviewer: Petabyte-scale, thousands of tables. The orders fact alone is billions of rows. Freshness ranges from minutes for streaming writes to daily for batch.

Candidate: Why not just keep the warehouse, or just the lake?

Interviewer: The warehouse is costly and closed - the ML team can't get cheap file access and we're locked to a vendor. The bare lake has none of the guarantees above. We want the lake's economics and openness with the warehouse's correctness.

Candidate: I'll assume the catalog can be a managed service and that query-engine internals are out of scope - we're designing the table layer, the catalog contract, and the maintenance that keeps it healthy.

Interviewer: Correct.

That pins it down.

Functional requirements

• Store large tables on object storage in an open format that multiple engines (Spark, Trino, Flink) read and write directly.
• ACID commits: a reader never sees a partial write, and a failed write leaves no trace.
• Support concurrent writers (streaming upsert plus batch) without corruption or lost updates.
• Schema evolution (add, rename, reorder, drop columns) and partition evolution, without rewriting existing data.
• Time travel to and rollback to a prior snapshot.
• Row-level updates and deletes (for CDC upserts and deletion requests).