The Design Framework: A Spine for Any Data Problem
Second attempt. Dev walks into another data round, and this time the prompt is "design analytics for our orders." A year ago he'd have started drawing. Today he asks: "Who reads these analytics, and how fresh do they need to be? Are we talking a daily revenue dashboard, or fraud alerts in seconds?" The interviewer leans back. The room relaxes. Dev hasn't drawn a single box and he's already winning, because he's shown he knows that the same words ("orders analytics") describe five completely different systems.
The thing that changed isn't knowledge. It's that he has a spine to walk down, out loud, every time.
The spine
Eight steps, source to consumer. You walk them in order, narrating as you go. You won't spend equal time on each, but you touch all of them so nothing important falls through a crack.
- Requirements and scope. Who consumes this, how fresh, how correct, how big, what's the budget. Pin down what's explicitly out of scope. This is where you win or lose the round.
- Sources and data contracts. Where does the data come from, what's its shape, who owns it, and what's the agreement for when it changes.
- Ingestion. How data gets from the source into your world. Batch pull, CDC, an event stream. This choice sets your freshness ceiling.
- Storage and layout. Where it lands and in what physical shape. File format, partitioning, table format, the bronze/silver/gold layering.
- Processing. How raw becomes useful. Batch transforms, stream processing, the joins and aggregations.
- Serving. How consumers actually read it. A warehouse, an OLAP engine, a key-value store for low-latency lookups, an API.
- Governance and SLA. Freshness guarantees, data quality gates, PII handling, lineage, who gets paged when it breaks.
- Failure modes. The 3am path. What happens on a double-send, a schema change, a skewed key, a replay of last quarter.
You don't have to use these exact labels in the room. You do have to cover this ground. When you feel yourself stalling, the fix is always "what's the next step on the spine?"
The two axes that decide every architecture
Strip away the logos and almost every data architecture decision lives on two axes. Get these two right and the technology choices mostly fall out.
Freshness: how soon after an event happens must it show up downstream?
batch (hours) → micro-batch (minutes) → streaming (seconds)
Correctness: what delivery guarantee does the consumer actually need?
at-most-once → at-least-once → exactly-once
Here's the part people miss: freshness and correctness both cost money, and they often pull against each other. Streaming-plus-exactly-once is the most expensive corner of the grid. Most systems don't need it. A revenue dashboard is happy with "batch, at-least-once, reconciled nightly." A billing counter needs "streaming, effectively exactly-once." Your job is to place the problem on the grid honestly, not to reach for the priciest corner because it sounds impressive.
A trick that reads as senior: ask "what's the cost of being a little stale here, and what's the cost of being a little wrong?" The answers put you on the grid instantly. A marketing funnel can be an hour stale and slightly wrong. A payment ledger can be neither.
The scoping questions that buy you the whole round
Step 1 is worth more than the other seven combined, because a wrong scope makes every later decision wrong. Spend real time here. The questions that consistently pay off:
- Who consumes this, and for what decision? A human glancing at a dashboard tolerates very different freshness than an automated system blocking a transaction.
- How fresh? Get a number or a range. "Daily" and "under a minute" are different products.
- How correct? Can a number be approximate, or must it reconcile to the penny? Approximate is dramatically cheaper.
- How much data? Rough volume and growth. This decides batch vs streaming feasibility and cost. (The next lesson is entirely about estimating this.)
- What are the sources, and who owns them? One clean Postgres table is a different world than 200 third-party feeds.
- What's the access pattern on the output? Big scans and aggregations, or point lookups by key? This picks your serving layer.
- What's the budget and the team? A two-person team should not be handed a system that needs a streaming platform babysat 24/7.
You won't get crisp answers to all of these. That's fine. Asking them is the signal. It shows you know that "design X" is underspecified until you've nailed these down.
Spend the 45 minutes like this
A rough budget so you don't run out of time with the interesting part undrawn:
| Phase | Time | What you're doing |
|---|---|---|
| Scope | ~5 min | The questions above. Write the requirements where you both can see them. |
| Estimate | ~5 min | Volume, velocity, storage, a rough cost. Enough to justify the architecture. |
| High-level design | ~10 min | One diagram, source to serving. Name components, not just vendors. |
| Deep dives | ~20 min | The 3 to 4 parts that actually get interrogated. This is where the points are. |
| Wrap-up | ~5 min | Failure modes, the tradeoff you're accepting, what you'd do with more time. |
The most common failure is spending twenty minutes on the high-level diagram and then having no time for a deep dive. The deep dives are where seniority shows. Draw the high level fast, then go deep on the part that's actually hard.
The pattern names worth knowing
When you place a problem on the freshness/correctness grid, a few named patterns recur. You don't have to use them, but naming the right one quickly signals fluency:
- Lambda. Two paths: a streaming "speed layer" for fast-but-rough answers and a batch layer that recomputes the correct answer and overwrites. Self-healing, but you maintain two codebases with the same logic. Reach for it when you need both low latency and provable correctness, like ad-click billing.
- Kappa. One streaming path, no separate batch layer. Reprocess by replaying the log through the same code. Simpler to maintain, but replaying years of history through a stream engine can be slower and pricier than a batch job over files.
- Medallion (bronze/silver/gold). Not about batch vs stream. It's a layering discipline: raw landing (bronze), cleaned/conformed (silver), business-ready aggregates (gold). Almost every lakehouse design uses this layout regardless of freshness.
- CDC. Turn a database into a change stream instead of repeatedly querying it. The default way to get fresh data out of an operational DB without hammering it.
- Data Mesh. An org pattern more than a tech one: domain teams own their data as products, with central standards. Comes up when the question is about 500 sources and who's responsible for them.
The framework's job isn't to hand you the answer. It's to keep you asking the next right question out loud, in an order the interviewer can follow. A design round is graded on the reasoning you make visible, not the diagram you end up with. The spine is how you keep that reasoning flowing when your mind goes blank.
Walk it once, fast
Take the prompt that started this lesson, "design analytics for PhoenixWorld's orders," and run the spine in two sentences per step:
- Scope: analysts want a daily revenue and cohort dashboard, yesterday-fresh is fine, must reconcile to finance. Real-time is out of scope.
- Sources and contracts: orders live in operational Postgres; we agree a contract that schema changes ship through a registry, not a surprise.
- Ingestion: a daily incremental pull on an updated-at watermark. Freshness need is low, so no streaming.
- Storage: land raw in Parquet on object storage, partitioned by date; model a star schema in the gold layer.
- Processing: a batch Spark job builds fact_orders and the customer dimension, with Type 2 history.
- Serving: a warehouse table the BI tool queries; pre-aggregate the common rollups.
- Governance and SLA: dashboard fresh by 7am, a freshness check that pages if the load is late, PII columns masked.
- Failure modes: late-arriving order updates, an idempotent reload so a rerun doesn't double-count, a backfill path for the day finance disputes.
That's a complete, defensible answer in under a minute of narration, and every later deep dive hangs off one of those steps. That's the whole point of having a spine. The next lesson sharpens step 2, the estimate, because "yesterday-fresh, reconcile to finance, daily pull" only becomes a real design once you know it's 50 GB a day and not 50 TB.